EDML for Learning Parameters in Directed and Undirected Graphical Models
EDML is a recently proposed algorithm for learning parameters in Bayesian networks. It was originally derived in terms of approximate inference on a metanetwork, which underlies the Bayesian approach to parameter estimation. While this initial derivation helped discover EDML in the first place and provided a concrete context for identifying some of its properties (e.g., in contrast to EM), the formal setting was somewhat tedious in the number of concepts it drew on. In this paper, we propose a greatly simplified perspective on EDML, which casts it as a general approach to continuous optimization. The new perspective has several advantages. First, it makes immediate some results that were non-trivial to prove initially. Second, it facilitates the design of EDML algorithms for new graphical models, leading to a new algorithm for learning parameters in Markov networks. We derive this algorithm in this paper, and show, empirically, that it can sometimes learn estimates more efficiently from complete data, compared to commonly used optimization methods, such as conjugate gradient and L-BFGS.

Summary: Overall, I think the paper is potentially
      useful and a contribution to the community since it puts forth a simpler
      framework for EDML (the previous works on EDML are quite dense). However,
      I was disappointed that it presented it in a confusing manner and that it
      didn't address a number of key issues such as convergence, relationship to
      EM, etc. Further, the contribution to parameter estimation for undirected
      models is quite incremental. Contribution: This paper portrays
      EDML as fixed-point iteration where fixed points correspond to stationary
      points of the log likelihood. Each iteration requires performing inference
      and has a notion of soft evidence, like EM. Their application to
      complete-data learning for Markov networks reduces to some Jacobi-style/
      coordinate descent-type method for the convex log likelihood and can be
      used as a black-box optimizer for Markov networks. Novelty:
      The connection between EDML and continuous optimization is novel.
      However, the literature on alternatives to EM is vast, and I'm not sure
      how novel it is as a learning method under missing data. For complete-data
      Markov networks, the method is similar to basic, known methods for convex
      optimization. Pros: Attractive method for learning in the
      presence of missing data that points to fertile ground for future work.
      Unified treatment of directed and undirected graphical models is also
      interesting and potentially impactful. Many practitioners would benefit
      from reading the paper in order to think about alternative perspectives on
      parameter learning in the presence of unobserved data. Cons:
      1) The paper casts EDML as an optimization technique, but lacks
      necessary details for readers to understand this interpretation. 2)
      Exposition, particularly notation, is sometimes confusing. 3)
      insufficient conceptual or experimental comparison to EM. 4)
      experiments only address an overly simple case of parameter learning
      problems (complete data). 1) I found section 2 very confusing. It
      first appears as though you're introducing some new family of optimization
      algorithms, but then it becomes apparent that you're discussing the
      general family of techniques described in the first paragraph of the
      related work section. I would reshape this section to first state that
      such a family exists and discuss their convergence properties. Then, I'd
      state that for the particular class of problems you're dealing with, these
      coordinate-wise minimizations are tractable. Claim1 is boring, and more
      space should be made for previewing the sort of content discussed in the
      first paragraph of the related work section. The paper also lacks
      a formal discussion of convergence behavior for the method. How do we know
      that the update rule at the top of page 5 will converge? I agree that its
      fixed points are fixed points of the marginal likelihood, but how do we
      know it will ever converge to one of these? This issue would be remedied
      if you were much more straightforward in section 2 about what you need for
      such coordinate minimization techniques to converge. Fixed point iteration
      is not guaranteed to converge, you need some sort of contraction. See
      proofs, for example, about the convergence of loopy belief propagation.
      Why does section 3.4 have the title 'Finding the Unique Optimum?"
      The subfunctions are convex, so each of them has a unique minimizer.
      However, the overall problem doesn't have a unique stationary point, and
      the algorithm you present in this section is for finding such a point. I
      understand that the updates at the top of page 5 are obtained by
      analytically minimizing simultaneously in every coordinate (where you find
      a unique minimum), but the title is very confusing. 2) The
      notation in section 3.2 (soft evidence) needs to be defined much more
      clearly. In equation (3) is there implicitly a sum over possible values
      that X_i can take on? I don't understand how you can get a marginal
      likelihood without summing over the possible values for each X_i. What's
      the difference between P(\eta_i|x) and P(X_i|x)? The corresponding
      appendix does not clear things up either. Furthermore, I didn't find 3.2
      particularly illuminating about the method without some comparison to
      'soft evidence' in EM. Also, it's strange that you present an
      algorithm at the top of page 5 based on an earlier work and leave it to
      the reader to deduce that this can also be obtained by applying the
      procedure from page 2 to the log likelihood. You should, at the very
      least, say that the updates at the top of page 5 correspond to exact
      simultaneous minimization of the subfunctions. You should also discuss
      under what conditions such fixed point iteration will converge. 3)
      In general, I'd like to see much more discussion about the relationship
      between EDML and EM. EM has some desirable properties (monotonicity of the
      updates) and some downsides (it's maximizing a concave lower bound on the
      marginal likelihood rather than the actual likelihood). It would be good
      to juxtapose these with the properties of EDML. Of course it would have
      been nice to see experiments comparing the two as well. 4) The
      experiments section only considers complete-data learning, in which the
      log likelihood is convex. In this setting EDML is just another convex
      optimization method, so a much more rigorous discussion of its convergence
      behavior, memory requirements, per-iteration complexity, robustness to
      errors in approximate inference, etc. is necessary. For future work, I
      would make sure to do experiments with missing data and compare to EM. The
      two methods are not guaranteed to converge to the same value of the
      marginal likelihood, so comparing solution quality, and not just time, is
      necessary. Minor points What exactly is the semantics of your
      feasibility function g(y)? Are you always assuming it's a projection onto
      the feasible set? Is the feasibility function you use in If you don't, g()
      isn't guaranteed to give you what you want. First, you make it sound like
      a projection function onto a feasible set, but then you use it as an
      indicator function for the set? You should define somewhere early on
      what EDML stands for. Is this technique stable if doing exact
      inference in the bayes net is intractable? Discussion or experimental
      evaluation in this setting would be useful. The update equation at the
      top of page 5 should have a number. ----- Post Author Response
      ----- In their response, the authors address many of the key omissions
      in their paper, such as the lack of a proof of convergence, by referring
      us to their other recent papers on EDML. This isn't quite sufficient to
      address the confusion that will arise for future readers of the paper.
      Some things, such as a comparison to EM can be addressed by providing a
      pointer to the earlier papers. However, the discussion of EDML as an
      optimization technique needs to be able to stand on its own. It is not
      sufficient to present an optimization algorithm and not prove that it
      converges or characterize what sorts of points it will converge to.
      Overall, I am familiar with the modern literature on inference and
      learning in graphical models, and this paper is not of sufficient quality
      to be presented at NIPS. My concerns with this paper are mostly
      presentation related. For subsequent versions of the paper, I hope the
      authors will address the concerns that were raised. This is quite an
      important contribution, and I hope authors will resubmit it again (if it
      is rejected at NIPS).


Adaptive Multi-Column Deep Neural Networks with Application to Robust Image Denoising
Stacked sparse denoising autoencoders (SSDAs) have recently been shown to be successful at removing noise from corrupted images. However, like most denoising techniques, the SSDA is not robust to variation in noise types beyond what it has seen during training. To address this limitation, we present the adaptive multi-column stacked sparse denoising autoencoder (AMC-SSDA), a novel technique of combining multiple SSDAs by (1) computing optimal column weights via solving a nonlinear optimization program and (2) training a separate network to predict the optimal weights. We eliminate the need to determine the type of noise, let alone its statistics, at test time and even show that the system can be robust to noise not seen in the training set. We show that state-of-the-art denoising performance can be achieved with a single system on a variety of different noise types. Additionally, we demonstrate the efficacy of AMC-SSDA as a preprocessing (denoising) algorithm by achieving strong classification performance on corrupted MNIST digits.

This paper uses deep neural networks to denoise
      images. The network uses several independent columns trained on different
      types of noise. Furthermore, a weight prediction module is trained to
      decide how to merge these columns to predict the clean image from the
      noisy image. The idea is to have only one network for all types of noises
      and noise strengths. Noise is not always Gaussian, and it may
      sometimes be beneficial to have a single method that can adapt to
      different noise types. Engineered methods like BM3D are hard to modify to
      arbitrary noise types, learned methods like Burger et al. have to be
      retrained (or also the missing citation Jancsary et al., "Loss-Speci fic
      Training of Non-Parametric Image Restoration Models"). This method
      solves it by training different models on different noises and then
      combining the results. The method is shown to be successful in denoising
      different types of noises after a one-time training. Denoising also works
      on "unseen" noise that wasn't trained on in any of its submodels.
      However, it is inferior to just training on a single noise type for
      that specific noise type: SSDA outperforms MC-SSDA on the CT images on
      Gaussian noise type 2 and 3, which have similar PSNR to the training PSNR
      of SSDA. The same holds true for the MNIST comparison. Furthermore, the
      SSDA is also able to denoise unseen noise types to some extent, in one
      case even better. [The authors were able to address this issue during the
      rebuttal by training a new model with additional columns.] Also, it's
      not compared to standard denoising methods on natural images, which I
      would expect. On digital photos the noise usually is a mixture of Gaussian
      and Poisson noise, which the method should be able to handle. I wonder
      what the runtime of the method is (training/testing)? Also, how does the
      performance change if the number of autoencoders is varied? The
      paper is well structured and easy to understand. Very beneficial is that
      the code would be published after publication.


A simple example of Dirichlet process mixture inconsistency for the number of components
For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of clusters -that is, the posterior on the number of components represented in the observed data. However, it turns out that this posterior is not consistent -it does not concentrate at the true number of components. In this note, we give an elementary proof of this inconsistency in what is perhaps the simplest possible setting: a DPM with normal components of unit variance, applied to data from a "mixture" with one standard normal component. Further, we show that this example exhibits severe inconsistency: instead of going to 1, the posterior probability that there is one cluster converges (in probability) to 0.

The title of this paper is much like the paper itself:
      to-the-point, descriptive, and readable. "A simple example of Dirichlet
      process mixture inconsistency for the number of components" delivers on
      its promise by providing two easy-to-understand demonstrations of the
      severity of the problem of using Dirichlet process mixtures to estimate
      the number of components in a mixture model. The authors start by
      demonstrating that making such a component-cardinality estimate is
      widespread in the literature (and therefore a problem deserving of
      interest), briefly describe the Dirichlet process mixture (DPM) model
      (with particular emphasis on the popular normal likelihood case), and then
      demonstrate with a simple single-component mixture example how poorly
      estimation of component cardinality can go (their convincing answer: very
      poorly). Not only was the paper enjoyable to read but,
      refreshingly, didn't try to fit 20 pages of material into an 8 page limit.
      One potential criticism of this paper is that this result should
      be well-known in some sense in the community. It is a widely cited result
      that the number of components of a DPM grows logarithmically with the size
      of the data (and indeed the popularity of the Pitman-Yor process extension
      of the DPM is due to the different way in which the number of components
      grows: as an almost sure power of the size of the data rather than merely
      logarithmically). However, this paper is still necessary for two key
      reasons. (1) Existing results deal only obliquely with consistency in
      component cardinality, and this paper seems to be the first to prove
      component-cardinality inconsistency directly. (2) The authors make a case
      that the community, in spite of existing results, uses the DPM to estimate
      the ``true'' number of components in a data set. Since there are cases
      where the number of components in a mixture can indeed be expected to grow
      with the size of the data (cf. the HDP extension to LDA for topic
      modeling), the authors could make an even more compelling argument by
      highlighting in text and via their citations those cases in the literature
      where other authors use the DPM while assuming a fixed (i.e., not varying
      with the number of data points) number of clusters in a data set of any
      size. The authors might, in particular, make the distinction between the
      number of components in the DPM generative model (infinite, as on line 38,
      page 1) and the number of components "active" in a given data set. It is
      not clear to me that using the DPM to estimate the active number of
      components for a data set of size N, while assuming that a data set of
      greater size would exhibit more components, is immediately problematic;
      the real problem seems to be estimating a fixed, underlying number of
      components. Some care in describing this subtlety might be useful.
      Perhaps my one major concern is that at a number of times in the
      paper the authors make what appear to be unsupported claims by referring
      to seemingly already-completed future work---that is, work that is not in
      the current paper or cited. E.g., on line 46, page 1, "as we will show
      elsewhere" and line 66, page 2 "Further details will be provided
      elsewhere", and even the extensions to general alpha in Proposition 3.1
      and Theorem 4.1. It seems inappropriate to state results that are not
      proved in the current paper or elsewhere in existing published
      literature---except perhaps as proposed "future work" in the conclusion.
      There is no way to judge the merits of these results until they are given
      in full. Moreover, I think the paper stands on its own merits, and these
      further results are not strictly necessary (though they might be nice
      bonuses). Here are some more details: * For page 1, line 46, I suggest
      removing the sentence or giving the full result; the counterexamples
      provided either stand on their own or they don't. * Page 2, lines
      64-66 make some interesting claims but again do not seem to be supported
      by any citations or work in the current paper. One option is to simply add
      the sampler (and some timing and experimental results) to this paper.
      Another is to delete the sentences or move them to a "future work"
      discussion. I agree that it's useful to point out that there are practical
      alternatives to the DPM. But part of what's problematic about these
      unsupported claims is that the reader can't tell whether many types of
      inference (including approximations that are not MCMC) are available or
      evaluate how the two MCMC samplers compare in speed, ease of use, etc.
      * In Proposition 3.1 and Theorem 4.1, I suggest providing a full proof
      of the general alpha case in the supplementary material (or in the main
      text if it is indeed "trivial" as stated on page 4, line 167). I
      also have a number of minor comments: * Page 2, line 72: Perhaps say a
      bit more explicitly about why the DPM cannot be used as a heuristic. I
      assume the idea here is that, given a fixed number of underlying
      components, we expect the same amount of heterogeneity across a large
      range of data cardinalities but can also expect the DPM to return very
      different estimates of heterogeneity as data-size changes in these cases.
      * Page 2, line 101: Introduce alpha in the text as a hyperparameter of
      the model before using it. Notation for the CRP is inconsistent across
      different disciplines, so it would help to name alpha as the
      "concentration parameter" and give its allowable range of values. *
      Page 2: This is a nice, careful treatment of ordered vs. unordered
      partitions. * Page 3, line 126: The "component" vs "cluster"
      distinction is somewhat unclear. Consider drawing upon notation to
      clarify. * Page 4, line 107: Note that X_1, X_2, ... ~ N(0,1) is a
      different assumption than in the statement of Proposition 3.1. I believe
      the proof given here holds for the more general case of the Proposition
      3.1 statement though. Interestingly, it seems to hold when X_1, X_2, etc
      are iid from a single point mass. If indeed true, that seems like a
      compelling special case that may be worth mentioning. * Page 6, line
      304: Consider making this remark about p(A) = 1/n in this special case the
      first time it is used instead (on page 4, line 192). ========
      Update after rebuttal: I have read the author response, which
      addressed my few and minor concerns. I look forward to reading the final
      product.


Training and Analyzing Deep Recurrent Neural Networks
Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this paper we study the effect of a hierarchy of recurrent neural networks on processing time series. Here, each layer is a recurrent network which receives the hidden state of the previous layer as input. This architecture allows us to perform hierarchical processing on difficult temporal tasks, and more naturally capture the structure of time series. We show that they reach state-of-the-art performance for recurrent networks in character-level language modeling when trained with simple stochastic gradient descent. We also offer an analysis of the different emergent time scales.

I like the point about the arbitrariness of older data
      benefiting from potentially many more layer of processing compared to
      newer data. And building a deep RNN seems like a reasonable way to do
      better on shorter term stuff, in addition to possible other benefits.
      Another obvious thing to do would be to put extra layers of nonlinearities
      between each timestep. Have you tried this? How important was the
      gradient normalizations and why does this "avoid of the problem of
      bifurcations"? There is a recent paper in ICML 2013 by researchers from U
      of Montreal that looks at gradient truncation and optimization in RNNs
      that that may be relevant here. Also, will normalizing the gradient in
      this way potentially mess up the "averaging" behavior of SGD? In
      terms of previous results on these kinds of Wikipedia compression tasks,
      there is also some work by Mikolov et al. that you may want to compare to.
      Of the various experiments designed to examine the different roles
      played by each layer in terms of time-scale and perhaps "abstraction", the
      one I find most persuasive is the text generation one (as shown in Table
      2). However, as pointed out by the authors themselves in the paragraph on
      line 241, it may be problematic to interpret the effect of this kind of
      "brain surgery" on RNNs due to the complex interdependencies that may have
      developed between the outputs of the various layers. For me,
      the biggest missing piece of the empirical puzzle in this paper is the
      question of whether the higher layers are actually *better* at processing
      more abstract and long-term properties of the text than if the units were
      moved to the first layer. i.e. are they benefiting from the extra levels
      of processing that proceed them in a nontrivial way? That they happen to
      take on these seemingly more abstract and longer term roles after training
      is good but incomplete evidence that this is the case. I notice
      that the regular RNN seems to have less parameters in these experiments
      since 2*767^2 * 5 > 2119^2 (I'm counting both recurrent and inter-layer
      weight matrices, hence the multiplication by 2), so the comparison might
      be a bit unfair. A more convincing would be if the deeper RNNs did better
      than a standard RNN with the same number of parameters, or even better,
      the same number of *units*. Say a 2 layer DRNN versus such an RNN with the
      same number of units, where the different in the number of parameters
      wouldn't favor the RNN so much that the comparison would be rendered
      unfair. This would be strengthen the paper's claims a lot in my opinion.
      Also, instead of Figure 2, it would be better to have seen how
      well various depths of DRNN did on the benchmarks when trained from
      scratch, possibly with wider layers than their deeper counterparts to make
      up for the difference the numbers of parameters and/or units.
      Minor: - You should define DRNN-AO and IO in the text
      somewhere and not just in Figure 1.


A Gang of Bandits
Multi-armed bandit problems formalize the exploration-exploitation trade-offs arising in several industrially relevant applications, such as online advertisement and, more generally, recommendation systems. In many cases, however, these applications have a strong social component, whose integration in the bandit algorithm could lead to a dramatic performance increase. For instance, content may be served to a group of users by taking advantage of an underlying network of social relationships among them. In this paper, we introduce novel algorithmic approaches to the solution of such networked bandit problems. More specifically, we design and analyze a global recommendation strategy which allocates a bandit algorithm to each network node (user) and allows it to "share" signals (contexts and payoffs) with the neghboring nodes. We then derive two more scalable variants of this strategy based on different ways of clustering the graph nodes. We experimentally compare the algorithm and its variants to state-of-the-art methods for contextual bandits that do not use the relational information. Our experiments, carried out on synthetic and real-world datasets, show a consistent increase in prediction performance obtained by exploiting the network structure.

This paper considers a setting where several
      contextual linear bandits are connected by the underlying graph. The
      assumptions is the weight vectors of the respective nodes are smooth over
      the graph (are close for neighbors). This appears to a be a good and rich
      enough model for recommendations in social networks. The algorithm
      runs classical contextual linear bandits on each node (which node is being
      tried is determined by nature), but then shares this information with
      other nodes as well so they also update their model estimates and
      confidence widths appropriately via Laplacian. Unfortunately, the
      regret analysis provided is disappointing. The authors in fact provide the
      analysis for a different algorithm than they propose and evaluate in the
      experiments. This could be perhaps acceptable but I found no reason
      whatsoever over for this. There is no discussion, how CB vs. TCB would
      change the algorithm, its evaluation and the deployment. At least the
      difference between TCB and CB should be discussed better than one could
      use the results from [1]. It is therefore possible, that the TCB-kind
      algorithm does not do well in the experiments or that for CB-kind it is
      not possible to provide learning guarantees. Maybe it would be better
      just to not include such analysis. Regarding the current analysis,
      it is not clear how big the terms ln|M_t| and 2L can be. How do they
      behave with T and n? This is needed in order to get some insight from the
      provided regret bound. And how does the graph noise influence these
      quantities? Furthermore, typically [4,9] the analyses for linear
      bandits give at least a guideline to set the exploration parameter \alpha.
      Unfortunately, we do get this from the analysis provided. What is the
      dependence of the upper bound on alpha anyway? Is it hidden in some of the
      quantities mentioned above? The experiment on the other hand are
      treated very well. The descriptions are very detailed and well explained.
      It is also great that the authors performed the experiments on real data
      and described the practical details to make the experiments reproducible.
      Finally, L186 - L190 The paper mentions that through graph, the
      reward influences other nodes in the "lesser extent", "somehow
      informatively" and "similar updates" are desired. However I do not see a
      parameter for "somehow" or "similar" in the algorithm. Could that be done
      by regularizing L by a multiple of I instead of I. In the experimental
      section, the authors mention that the clustering acts as a regularizer.
      Would not regularizing the Laplacian instead be a more principled way
      to achieve this regularization? Other comments - Why it is
      L(u1,...un) in the regret bound and not the regularized version, since
      the Laplacian is regularized in the algorithm? After the rebuttal:
      While this is a good work, I believe that including the theoretical
      analysis of a changed algorithm with not entirely clear relationship
      between the two, makes it confusing. I would increase my score if the
      theoretical analysis was taken away.


Bayesian Inference and Learning in Gaussian Process State-Space Models with Particle MCMC
State-space models are successfully used in many areas of science, engineering and economics to model time series and dynamical systems. We present a fully Bayesian approach to inference and learning (i.e. state estimation and system identification) in nonlinear nonparametric state-space models. We place a Gaussian process prior over the state transition dynamics, resulting in a flexible model able to capture complex dynamical phenomena. To enable efficient inference, we marginalize over the transition dynamics function and, instead, infer directly the joint smoothing distribution using specially tailored Particle Markov Chain Monte Carlo samplers. Once a sample from the smoothing distribution is computed, the state transition predictive distribution can be formulated analytically. Our approach preserves the full nonparametric expressivity of the model and can make use of sparse Gaussian processes to greatly reduce computational complexity.

This paper presents a Bayesian approach to state and
      parameter estimation in nonlinear state-space models, while also learning
      the transition dynamics through the use of a Gaussian process (GP) prior.
      The inference mechanism is based on particle Markov chain Monte Carlo
      (PMCMC) with the recently-introduced idea of ancestor sampling. The paper
      also discusses computational efficiencies to be had with respect to
      sparsity and low-rank Cholesky updates. This is a technically
      sound and strong paper with clear and accessible presentation. The online
      marginalisation of the transition dynamics and the use of ancestor
      sampling to achieve this is novel. The consideration of computational
      issues such as sparsity and low-rank updates/downdates to the Cholesky
      factors of covariance matrices strengthens the paper further. The
      empirical results, while brief, are sufficient (further suggestions
      below). In addition to its stated aims, the paper will likely
      stimulate discussion around inference methods for non-Markovian
      state-space models and the potential advantages/disadvantages of learning
      the transition dynamics in this way rather than specifying a parametric
      model a priori. While space is slight, the authors may like to
      consider some further discussion around the differences between using a
      parametric transition model given a priori against the use of a similar
      model as the mean function of the GP. For example in out of sample
      prediction (e.g. forecasting). The results in Table 1 and the
      description in the preceding paragraph are slightly unclear to me. I am
      unsure as to whether the RMSE is against a withheld set of data points or
      the same set of data points that is conditioned upon (the *|data in the
      column headings). My main interest would be an RMSE against an
      out-of-sample prediction, especially a forecast forward in time against
      withheld data. It is in this scenario that I would expect to see the
      largest differences between the learnt dynamics and the ground truth
      model. If Table 1 is not already showing this, an extra column that does
      so would be a great addition. One minor point: the abbreviation
      CPF-AS is used in Algorithm 1 before being defined in the first paragraph
      of Section 3.3.1 below.


Point Based Value Iteration with Optimal Belief Compression for Dec-POMDPs
We present four major results towards solving decentralized partially observable Markov decision problems (DecPOMDPs) culminating in an algorithm that outperforms all existing algorithms on all but one standard infinite-horizon benchmark problems. (1) We give an integer program that solves collaborative Bayesian games (CBGs). The program is notable because its linear relaxation is very often integral. (2) We show that a DecPOMDP with bounded belief can be converted to a POMDP (albeit with actions exponential in the number of beliefs). These actions correspond to strategies of a CBG. (3) We present a method to transform any DecPOMDP into a DecPOMDP with bounded beliefs (the number of beliefs is a free parameter) using optimal (not lossless) belief compression. (4) We show that the combination of these results opens the door for new classes of DecPOMDP algorithms based on previous POMDP algorithms. We choose one such algorithm, point-based valued iteration, and modify it to produce the first tractable value iteration method for DecPOMDPs that outperforms existing algorithms.

SUMMARY: This paper proposes a new method for
      Dec-POMDP planning that is built out of several components. The first is a
      new way of solving cooperative Bayesian games using an integer linear
      program. The second is the transformation of the Dec-POMDP to a belief
      POMDP in which a "centralized mediator" must select at each timestep the
      best action for each agent-belief pair. The third is to automate the
      discovery of optimal belief compression by dividing each timestep into two
      parts, the first corresponding to the original Dec-POMDP and the second
      giving each agent a chance to select how its beliefs in that timestep are
      mapped to a bounded set and thus compressed. The fourth assembles these
      components together into a point-based value iteration method that solves
      the resulting belief POMDP using a varient of PERSEUS in which the CBG
      solver is used to compute maximizations. ORIGINALITY: To
      my knowledge, the integer linear programming formulation of the CBG is
      novel, as is the optimal belief compression scheme. The latter in
      particular is a very original and intriguing idea. The PBVI method is also
      novel as far as I know, though it is only a minor variation on PERSEUS.
      However, I have serious doubts about the originality of the belief POMDP
      formulation presented in Section 4. In particular, it bears a striking
      similarity to the continuous-state MDP approach recently proposed by
      Dibangoye et al. [1]. Though those authors use an MDP formulation, this is
      really just the belief MDP of a POMDP that, as far as I can tell, is
      conceptually identical to the belief POMDP formulation used in this paper.
      In Dibangoye et al., the actions correspond to assignments of decision
      rules to each agent, such that the central planner has a Markov state
      representation but each decision rule conditions only on individual
      observations available to each agent. This is exactly the approach taken
      here as well. Section 4 may also relate to recent work by Oliehoek [2],
      though the exact nature of the relationship is less clear to me in that
      case. The fact that this submission doesn't compare to or even cite these
      highly related papers is a significant concern for me. CLARITY:
      Overall this is a polished and well written work. I found some
      aspects of the exposition unclear (see detailed comments below) but these
      were not fatal. SIGNIFICANCE: Given the questions about
      novelty of Section 4, the algorithmic contribution is moderate: somewhat
      incremental but certainly nontrivial. In my opinion, the most interesting
      and potentially significant aspect is the belief compression approach of
      Section 5. However, I am also somewhat skeptical that this idea is
      practical. Intuitively, it seems that there must be a trade-off: asking
      the planner to determine how best to compress beliefs must increase
      computational costs that might or might not outweigh the benefit of
      compressing beliefs. Unfortunately, no analysis (or even mention) of this
      trade-off appears in the paper. Furthermore, the empirical analysis does
      not isolate the effect of this component of the algorithm. This is an
      example of a more general problem limiting the paper's significance.
      Because the experiments only assess the complete system, no insights are
      gleaned into how each component contributes to performance. Does it work
      because of or in spite of each component? Is the total performance gain
      more than the sum of that of its parts? The paper would be greatly
      improved if it addressed such questions. Detailed comments:
      1: PERSUES -> PERSEUS 3: The explanation of the integer
      linear program is unclear, which I believe is due to confusion between
      state and joint observations. The variable x_a\theta is indexed by \theta
      but then we are told there is one for each state. (2) contains no mention
      of state but the text below it refers to feasible assignments of x_as.
      Also, constraints 2 -> constraints (2). 4: I don't understand
      why the assumption there is a bounded number of possible beliefs is
      important. First of all, it seems to me that in any finite Dec-POMDP,
      there is at any timestep only a finite number of reachable beliefs. The
      fact that this number can increase over time would mean that the size of
      the action space of the belief POMDP grows over time, but I don't see why
      that's problematic for the formulation. I can understand bounding the
      number of beliefs as a compression technique to yield efficient
      approximations, but this bound is presented as a prerequisite for the
      conversion to a belief POMDP which doesn't seem right to me. Also, the use
      of the word "observation" is confusing in this section. How can there be a
      one-to-one mapping between observations and beliefs? Do you mean
      observation history? Similarly, we are told on line 173 that there are no
      observations in the belief POMDP but then told in line 178 that actions in
      this belief POMDP are mappings from observations to actions. Why is the
      Lemma numbered 4.0.1? 5: Is t_i the same as |B|? 6: What
      is a concave and locally linear value vector? Aren't these vectors
      piecewise linear and convex? 7: To what horizon where these
      problems tested? How many hours were used by the methods that obtained the
      "Previous Best Utility"? If it's not the same as for your PBVI method, how
      can the comparison be fair? Refs: Is [18] correct? It has the same
      title and authors as [17]. [1] Optimally Solving Dec-POMDPs as
      Continuous-State MDPs. Jilles S. Dibangoye, Christopher Amato, Olivier
      Buffet and François Charpillet. In Proceedings of the Twenty-Third
      International Joint Conference on Artificial Intelligence (IJCAI-13),
      2013. [2] Frans A. Oliehoek. Sufficient Plan-Time Statistics for
      Decentralized POMDPs. In Proceedings of the Twenty-Third International
      Joint Conference on Artificial Intelligence, 2013.


Neural representation of action sequences: how far can a simple snippet-matching model take us?
The macaque Superior Temporal Sulcus (STS) is a brain area that receives and integrates inputs from both the ventral and dorsal visual processing streams (thought to specialize in form and motion processing respectively). For the processing of articulated actions, prior work has shown that even a small population of STS neurons contains sufficient information for the decoding of actor invariant to action, action invariant to actor, as well as the specific conjunction of actor and action. This paper addresses two questions. First, what are the invariance properties of individual neural representations (rather than the population representation) in STS? Second, what are the neural encoding mechanisms that can produce such individual neural representations from streams of pixel images? We find that a simple model, one that simply computes a linear weighted sum of ventral and dorsal responses to short action "snippets", produces surprisingly good fits to the neural data. Interestingly, even using inputs from a single stream, both actor-invariance and action-invariance can be accounted for, by having different linear weights.

SUMMARY This paper makes two points. First
      individual neurons in superior temporal sulcus have a continuous and
      distributed encoding of both the actor and action (and their conjunction)
      during action observation. It then makes the second point that the evoked
      responses (mean firing rate) can be predicted by a linear mixture of
      transformed pixel input – where the transform is motivated by existing
      models of dorsal and ventral stream processing. The report is written
      extremely nicely; however, it is rather descriptive and could be improved
      with some clarification of its conceptual direction and analysis details.
      COMMENTS TO AUTHORS Your report was written very
      nicely and enjoyable to read. On the other hand, there was a slight
      ambiguity about the underlying message and the connection between the
      snippet-matching model and your analyses. Perhaps you could consider the
      following; 1) Your analyses make two separate points. First, there
      are action and actor invariant responses in STS neurons. Second, you can
      predict the responses of STS neurons using a transformation of pixel
      inputs which – at the last stage – involves a linear mixture. My problem
      here is that these two points are unconnected. You could have presented
      the invariance analyses separately from the linear mixture analyses and
      vice versa. This is not necessarily bad but I think you need to clarify
      what the contribution of this report is and try to link the two points
      through some contribution to our understanding of perceptual synthesis?
      2) It is not clear how your linear regression analysis relates to
      the snippet-matching model. Your discussion of the snippet-matching model
      could be interpreted as either a response to a short sequence of inputs
      (with explicit temporal support) or simply a response to a time average
      (integrating the input over a small time window). Given that you seem to
      interpret your regression analysis in reference to the snippet-matching
      model, I suspect the latter is the case. If this is so, could you make it
      clear that each time bin or time step in the regression analysis
      corresponds to a snippet of input and that this is summarized by the
      average over 120 milliseconds. Furthermore, say how many time steps there
      were – in other words quantify T. It is confusing because you introduce
      snippet-matching after the regression analysis and then say: “Again, we
      try the simplest possibly function:” When you say “Again”, does this mean
      you have done two analyses and have only reported one or are you simply
      interpreting regression analysis in terms of the snippet-matching model.
      If the latter, it might be useful to introduce the snippet-matching model
      first and then describe how it motivated your linear regression.
      MINOR POINTS 1) In the abstract, you need to make it clear
      that this paper is about characterising neuronal responses to action
      observation. The title and first sentences made me think I was going to
      read about motor control. 2) I am not sure about your argument in
      the second paragraph of the introduction. Simply showing that there is
      information about actions, actors (and their conjunction) in any part of
      the brain does not mean they have tackled a “(three-fold) challenge”. One
      would find (presumably) very similar response properties in any part of
      the mirror neuron system. Furthermore, if one analysed retinal cells, one
      would also find this information. Perhaps you could highlight the fact
      that you have found information or invariance properties at the level of
      the single neuron - that could not be found at low levels in the visual
      hierarchy - to make your point more clearly? 3) Your use of cross
      validation to establish the utility of the linear regression model of
      neuronal responses is fine for the machine learning community. However, it
      is statistically sub-optimal in relation to inference based directly on
      your general linear model: by central limit theorem, you can assume
      Gaussian errors and provide a much more sensitive analysis in terms of
      classical inference. Furthermore, you could have used conventional F
      statistics to test different hypotheses about the relative contribution of
      dorsal and ventral predictors of the neuronal responses – or indeed their
      interaction. I suspect you will choose to stay with the cross validation
      scheme but it might be interesting to pursue standard statistical analyses
      in future work? 4) On page 5 (line 250) I would say “A more
      continuous and distributed representation scheme”. 5) On page 6
      (line 299) what is “a matching function”? The notion of a snippet-matching
      process is not described very clearly. Is this simply an instantaneous
      mapping between some running average? Or is there something deeper going
      on? 6) Page 5 (line 304), I would spell out that the dorsal and
      ventral predictors both had a correlation coefficient of 0.33 – otherwise,
      people will think this is the correlation coefficient for the combined
      analysis. 7) You do not report any statistical inference or
      p-values. It might be nice to associate p-values with your correlation
      coefficients and use a Fishers transform when comparing the correlations
      between different regression models (for example, the processed versus
      non-processed principal components). I hope that these comments
      help should any revision be required.


Σ-Optimality for Active Learning on Gaussian Random Fields
A common classifier for unlabeled nodes on undirected graphs uses label propagation from the labeled nodes, equivalent to the harmonic predictor on Gaussian random fields (GRFs). For active learning on GRFs, the commonly used V-optimality criterion queries nodes that reduce the L 2 (regression) loss. V-optimality satisfies a submodularity property showing that greedy reduction produces a (1 − 1/e) globally optimal solution. However, L 2 loss may not characterise the true nature of 0/1 loss in classification problems and thus may not be the best choice for active learning. We consider a new criterion we call Σ-optimality, which queries the node that minimizes the sum of the elements in the predictive covariance. Σ-optimality directly optimizes the risk of the surveying problem, which is to determine the proportion of nodes belonging to one class. In this paper we extend submodularity guarantees from V-optimality to Σ-optimality using properties specific to GRFs. We further show that GRFs satisfy the suppressor-free condition in addition to the conditional independence inherited from Markov random fields. We test Σ-optimality on real-world graphs with both synthetic and real data and show that it outperforms V-optimality and other related methods on classification.

The authors suggest the use of a criterion,
      Σ-Optimality, for active learning in Gauss-Markov random fields. The
      criterion itself was originally proposed by Garnett et al for active
      surveying, but it does not appear that the submodular property was
      recognized in that previous work. Labeled and unlabeled are
      embedded in a graph nodes represent both labeled and unlabled data and
      edge weights, computed via a kernel, capture similarity. The
      motivation for an active approach is that acquiring labels on the full
      data set may incur some cost (presumably greater than computing the
      edge weights over all data) so a criterion is used to determine which
      of the remaining unlabeled data should be labeled. The authors
      establish that the criterion satifies the submodular monotone property
      and as such greedy selection achieve (1-1/e) performance relative to
      optimal selection. The authors should be careful to note that this
      optimality is with respect to the criterion itself and not with
      respect to classification accuracy. While the empirical results do
      give good classification performance, the criterion itself is only a
      surrogate. Several existing criterion are mentioned in the
      introduction (Settles, Krause et al, Ji and Han) to which the authors
      compare their criterion. One item which is only given limited
      discusion is the computational complexity of computing the reward. For
      example, both V-optimality and Sigma-optimality require computing the
      inverse of the graph Laplacian over the remaining unlabeled data. Some
      of the other criterion have lower complexity. This is an important
      issue since the problem is cast in terms of costs and since
      performance of all criterion will eventually approach each other
      (since they are presumably solving the same classification problem
      following selection) a fairer comparison would include the cost of
      computing the reward. It may be that the proposed method wins here,
      but it at least bears some mention. Why do the authors cite
      Streeter and Golovin for results regarding submodular set functions?
      Shouldn't this be Nemhauser et al? Section 2.3 misstates several
      things. The intractability of subset selection is not a consequence of
      submodularity, it is simply a consequence of the combinatorics.
      Furthermore, this does not mean that a greedy solution is "required".
      Other strategies may have lower complexity and outperform the greedy
      selection. It is merely, that by establishing the property, greedy
      selection has certain guarantees relative to the optimal selection.
      The fact that the criterion of interest was originally proposed by
      Garnett et al in 2012 should be mentioned much earlier. This changes
      the abstract from "We propos a new criterion" to something more
      accurate, such as "We analyze a previously suggested criterion and
      demonstrate its utility...". Establishing the sumodular
      property as well as the suppressor free properties are interesting.
      Empirical results are suffcient. Modulo the comments
      above, the paper is fairly clear. The results are interesting and the
      analysis, though limited, represents a contribution. minor
      comments: Figure 3 lacks a horizontal axis. I assume it is
      probability of correct classification, but the authors give an
      ambiguous description in the caption.


Learning Kernels Using Local Rademacher Complexity
We use the notion of local Rademacher complexity to design new algorithms for learning kernels. Our algorithms thereby benefit from the sharper learning bounds based on that notion which, under certain general conditions, guarantee a faster convergence rate. We devise two new learning kernel algorithms: one based on a convex optimization problem for which we give an efficient solution using existing learning kernel techniques, and another one that can be formulated as a DC-programming problem for which we describe a solution in detail. We also report the results of experiments with both algorithms in both binary and multi-class classification tasks.

Note: References present in the paper are referred to
      by their numerical citations as used in the paper. Additional references
      are included as Natbib style citations. The complete citations are given
      at the end of the review. I have read the author rebuttal
      carefully and I would like to retain my original review and assessment.
      Summary ======================================== The paper
      addresses the problem of multiple kernel learning wherein a kernel K_\mu
      must be learned that is a positive combination of a set of M base kernels
      K_mu = \sum \mu_i K_i. The aim of this paper is to formulate the kernel
      learning problem in a way that yields tight learning theoretic guarantees.
      The motivation for the approach comes from some existing learning
      theoretic analyses for the kernel learning problem [12,18] that
      demonstrate that "fast" O(1/n)-style convergence guarantees can be
      achieved by analyzing the local Rademacher averages of the hypothesis
      class. These analyses indicate that it is the "tail sum" of the
      eigenvalues (sum of all but a few top eigenvalues) rather than the
      complete trace that decides the convergence properties. This leads
      the authors to formulate the kernel learning problem with "tail-sum"
      regularization instead of trace-norm regularization (the cut-off point for
      the tail being a tunable parameter). However, the resulting class H1 turns
      out to be concave which prompts the authors to propose two learning
      formulations 1) Algorithm 1: A convexification of the class H1 is
      proposed that yields a class H2 that instead regularizes using a lower
      bound on the tail sum of eigenvalues of the combined kernel. For this
      class, existing L1-norm MKL solvers are applicable and are used.
      2) Algorithm 2: To work with the concave class H2 itself, the
      authors propose a DC programming approach wherein at each stage, the
      concave constraint is linearized and the resultant (convex) problem is
      solved via an alternating approach. Some interesting properties of
      the two classes H1 and H2 are analyzed as well. For instance, it is shown
      that H2 is not simply the convex closure of H1. Generalization guarantees
      are given for both learning formulations with the excess risk behaving as
      O(1/n) for base kernels with spectra that die down quickly (e.g. the
      Gaussian kernel). The approach, being motivated from learning theoretic
      considerations, has good generalization properties (which albeit follow
      rather painlessly from existing results). Experiments are
      conducted with standard L1, L2 as benchmarks. The trivial uniform
      combination of kernels is also taken as a benchmark. Performance
      improvements over these three benchmarks are reported on the transcription
      start site detection problem where both Algorithm 1 and 2 give moderate
      performance improvements (being the best on all training set sizes
      considered in terms of area under the ROC curve). Both Algorithm 1 and 2
      themselves give similar performance. It is also shown that the proposed
      approach can exploit moderately informative kernels which the L1 approach
      can discard. Experiments are also carried out on some multi-class
      datasets where performance improvements are reported for Algorithm 1 over
      the uniform kernel combination and standard L1 and L2 MKL formulations.
      Questions ---------------------------------------- 1)
      Theoretical results are given for the squared loss but the experiments use
      the standard hinge loss. Can this gap be bridged for a more complete
      presentation (in a future manuscript perhaps)? Hinge loss might be unable
      to guarantee fast rates unless low noise assumptions are made but still,
      any such analysis would give a more complete picture. Authors may also try
      kernelized ridge regression in experiments to see if it compares with
      hinge loss. 2) How is the cut-off parameter \theta chosen ? The
      experiments section mentions a model parameter h: is this treated as theta
      ? 3) How is the proposed approach different from one that simply
      requires the learned kernel to be low rank ? The tail-sum is minimized
      when it is zero in which case one is looking at a low rank kernel. Since
      minimizing rank is hard (even NP-hard in some cases), what the paper seems
      to be proposing is some relaxed version of rank (not trace norm though)
      that still encourages low rank). 4) The results of [8] were
      further improved in (Hussain and Shawe-Taylor, 2011). The authors may wish
      to reference this result. 5) It seems that the authors have
      modified the default inter-line spacing values (possibly by modifying the
      "baselinestretch" parameter in LaTeX). Since this is a notation heavy
      paper, this modification is making the paper look a bit crammed. Please
      revert to default line spacing as prescribed in the NIPS style file if
      modifications have indeed been made. 6) Line 208: ... two distinct
      learning kernel algorithm*s, * each based ... 7) Table 1 caption:
      ... either l1-norm or *l2*-norm regularization. 8) References are
      not in order in which they appear in the manuscript. Please use the unsrt
      bibliography style to implement this. 9) DC programming can be
      expensive: can some data be given so as to give an idea about the
      computational costs involved in the method ? Quality
      ======================================== This is a good paper with
      proper theoretical analysis as well as experimental work. Clarity
      ======================================== The paper is well written
      with expository appendices in the supplementary material.
      Originality ======================================== The
      idea of using tail sum regularization is new and throws up several
      questions since it results in a concave problem. On the algorithmic front,
      existing results are sufficient to implement the ideas.
      Significance ======================================== The
      paper proposes a new technique for designing multiple kernel learning
      algorithms that tries to formulate the learning problem in a way that will
      lead to good learning theoretic guarantees. This could have other
      potential applications. References
      ======================================== Zakria Hussain, John
      Shawe-Taylor: Improved Loss Bounds For Multiple Kernel Learning, AISTATS,
      JMLR W&CP, (15), pages 370-377, 2011.


Scalable Inference for Logistic-Normal Topic Models
Logistic-normal topic models can effectively discover correlation structures among latent topics. However, their inference remains a challenge because of the non-conjugacy between the logistic-normal prior and multinomial topic mixing proportions. Existing algorithms either make restricting mean-field assumptions or are not scalable to large-scale applications. This paper presents a partially collapsed Gibbs sampling algorithm that approaches the provably correct distribution by exploring the ideas of data augmentation. To improve time efficiency, we further present a parallel implementation that can deal with large-scale applications and learn the correlation structures of thousands of topics from millions of documents. Extensive empirical results demonstrate the promise.

The authors describe a two novel inference methods for
      the correlated topic model (CTM). They build on analytic results for
      the conditional logistic normal likelihood to arrive at a fast,
      easily parallelized exact inference. This leads to an approximate
      sampling method for producing Polya-Gamma variates. Finally, they
      propose a method for efficiently drawing samples in the presence of
      sparsity. To demonstrate the utility of the topic correlations
      inferred by the correlated topic model, the authors show a
      hierarchical clustering of 1000 topics learned from the 300k document
      New York Times dataset. While the figure makes a good case for the
      usefulness of these sort of correlations, it seems a bit out of place
      in the present paper, where the focus is on speeding up inference of
      an existing model as opposed to a suggestion of a new model. This page of
      the paper could have been better used further expanding on the details
      of the inference. To assess the effectiveness of their
      inference, the authors compare against a variational CTM inference on
      the New York Times dataset. The authors examine perplexity and
      training time of the variational method and both serial and parallel
      runs of their inference, in both cases finding little degradation in
      perplexity performance while showing up to two orders of magnitude
      faster training times. To show the scalability of the
      approach, the authors compare the training time and perplexity of
      their approach with that of Y!LDA, both running on 480 distributed
      processors. Here they demonstrate similar perplexity and while Y!LDA
      significantly outperforms CTM in training runtime, this must be
      weighed against the value of the correlation structure that CTM
      produces. The authors give a brief discussion to the effect of the
      various burn-in times of the inference and show that the priors have
      no measurable effect on perplexity. That being said, this is
      demonstrated only over one small corpus with 100 topics. How robust
      are these parameters to other corpus sizes and numbers of topics?
      Finally, the authors note that the training time of their
      parallel inference is nearly constant as document count and number of
      compute nodes are increased in proportion. This is a clear
      strength of the method. The authors put forth a compelling
      inference for a challenging model. They demonstrate prediction
      performance comparable to existing, but much slower, variational
      methods while significantly improving runtime. While the discussion of
      the inference itself could be expanded (especially in light of the
      page spent on Figure 3), the contribution is clear and enables the CTM
      to be used on datasets far larger than previously feasible.
      This paper comprises a solid contribution to the topic modeling
      community. So far, the slow inference of CTMs have prevented the community
      from building on it. This paper has the potential to change this. The
      derivation is sound and the evaluation is convincing.
      -------------------- Update of comments after the
      discussion period: In my understanding Y!LDA refers to a
      non-correlated version of LDA. I was impressed that this model is able to
      learn this more complicated model for a computational price that is
      comparable to Y!LDA, which is heavily tuned and less expressive. I
      prefer papers that attempt a new approach, even if they do not strictly
      outperform a heavily tuned baseline. I think this approach is worthwhile
      considering and may lead to further improvements as people pick the idea
      up and refine it. The fact that neither me nor
      Assigned_Reviewer_7 was not familiar the Polya-Gamma distribution is a
      sign that this knowledge should be promoted in the community, even if it
      is a straight forward application of work on Polya Gamma distributions. I
      would appreciate if relations to the paper of Polson, Scott and Windle,
      2013 would be clarified in the writeup. I have not seen many
      practical applications of the correlated topic model, and I think this is
      partially due to problems with the slow inference. I am hoping that this
      paper will motivate some new inference methods for this and other complex
      topic models, and that's why I would like to see it at the conference.


Optimization, Learning, and Games with Predictable Sequences
We provide several applications of Optimistic Mirror Descent, an online learning algorithm based on the idea of predictable sequences. First, we recover the Mirror Prox algorithm for offline optimization, prove an extension to Hölder-smooth functions, and apply the results to saddle-point type problems. Next, we prove that a version of Optimistic Mirror Descent (which has a close relation to the Exponential Weights algorithm) can be used by two strongly-uncoupled players in a finite zero-sum matrix game to converge to the minimax equilibrium at the rate of O((log T )T ). This addresses a question of Daskalakis et al [6]. Further, we consider a partial information version of the problem. We then apply the results to convex programming and exhibit a simple algorithm for the approximate Max Flow problem.

The authors apply the Optimistic Mirror Descent
      algorithm to a variety of problems where smoothness induces a certain
      amount of predictability in the resulting loss function sequences. In
      particular, better rates are possible when either: 1) the objective G is
      fixed for all t, and smooth, or 2) the problem is a saddle point problem
      on a function phi(f, x) which is appropriately smooth for each of its
      arguments individually, and the optimistic mirror descent algorithm is
      played against itself. These are rather specialized notions of
      predictability, but they lead to some interesting results. Section 4
      extends this to zero-sum games, and Section 5 considers approximate smooth
      convex programming. Unlike many applications of online algorithms,
      the goal in most cases in this paper is to solve a batch optimization
      problem. This should probably be made more explicit in the introduction.
      Section 3: The author’s need to be more clear about why this
      section improves on the result from Lemma 3. The key point (I think) is
      that the smoothness assumptions of Cor 5 may be satisfied even if G(f) =
      inf_x phi(f, x) is not smooth. Note that an alternative approach to
      solving the problem of Sec 3 is to run an online algorithm against a
      best-response adversary. It is worth asking whether or not any online
      algorithm paired with a best-response oracle can lead to the convergence
      rates like 1/T produced by the procedure introduced here. (Perhaps not,
      since this would appear to destroy predictability). Section 4: Be
      more clear on how this section improves on Section 3. The key seems to be
      adding robustness to an arbitrary adversary in addition to the good bounds
      for the predictable one. The results of Sec 4.2 are rather weaker than the
      section title suggests. I would suggest describing this setting as
      multi-point bandit feedback, and citing “Optimal Algorithms for Online
      Convex Optimization with Multi-Point Bandit Feedback”. Based on the
      notational similarities, it seems quite likely the authors are aware of
      this work. It really should have been cited in the submission. It also
      seems somewhat surprising that 4 points are needed, rather than just 2 ---
      this should be discussed. Section 5: It’s not clear to me that one
      can assume F^* is known WLOG. At the very least, the binary search
      procedure needs to be made precise: Are we running the algorithm of
      section 5 at each step of the binary search? If so, how do you check
      whether the guess of F^* is too high or too low? How does using a guess of
      F^* that is off by some amount delta impact the final bound of Lemma 8?
      This carries over to Sec 5.1 as well of course. Also, Eq (11)
      appears to fit into the setup of Sec 3 (Cor 5), why do we need the
      machinery of Sec 4.2 here (Lemma 7)? Is Line 368 a typo? In
      general, the paper suffers from some typos that made it more difficult to
      review, and could also probably use a bit more discussion of related work
      (several particular citations are recommended below). Local
      points: - Line 112: This simple intuition and analysis for Mirror
      Prox is very nice. - Line 93: This bound can be seen as a direct
      generalization of the result that the “Be The Leader” (BTL) algorithm
      suffers no regret, since you arrive at a BTL algorithm by taking Delta_t =
      M_t. This is worth mentioning, as well as a citation to “Efﬁcient
      algorithms for online decision problems” (Adam Kalai and Santosh Vempala).
      - Line 131: I would find the section title “Saddle-Point
      Optimization” more clear. - Line 154-155: citing “Adaptive game
      playing using multiplicative weights” (Yoav Freund and Robert E. Schapire)
      is probably appropriate. This citation is also appropriate at lines
      208-209. - Line 223: More detail needed, explain exactly why Sec 3
      doesn’t lead to this result (because it doesn’t handle an arbitrary
      adversary as well?), as well as why a direct application of mirror-prox
      does not provide strong decoupling. - Line 225, also lines 265 and
      268: It’s imprecise to refer to regret bounds like 1/T or 1/Sqrt(T), you
      need to multiply through by T to get regret as defined on line 65.
      Alternatively, refer to these as convergence rates. - Line
      312-314: Typos in both “Build estimates” sections of the pseudo-code, you
      want r^+ - r^-, as written all the a’s and b’s are zero. - Line
      366: Epsilon hasn’t be defined yet, introduce as an arbitrary constant eps
      > 0 earlier in the statement. - Line 368: Do you mean Corollary
      5, not Lemma 7? If Lemma 7 is really intended, then the procedure needs to
      be made more clear, e.g., what is the matrix A here? - Appendix:
      The proofs make extensive use of copy-paste, which makes them much harder
      to read because there is so much repetition. Please re-structure the
      proofs so they are less verbose and the main arguments are more
      clear.


Variational Inference for Mahalanobis Distance Metrics in Gaussian Process Regression
We introduce a novel variational method that allows to approximately integrate out kernel hyperparameters, such as length-scales, in Gaussian process regression. This approach consists of a novel variant of the variational framework that has been recently developed for the Gaussian process latent variable model which additionally makes use of a standardised representation of the Gaussian process. We consider this technique for learning Mahalanobis distance metrics in a Gaussian process regression setting and provide experimental evaluations and comparisons with existing methods by considering datasets with high-dimensional inputs.

The paper proposes a variational bound on the length
      scale parameters of square-exponential-kernel Gaussian process regression
      models. The main idea is to separate the function to be inferred into a
      "standardised" sample from a unit-length-scale square-exponential kernel,
      and a linear scaling map of that latent function, then to impose
      factorisation between these two objects via a variational bound.
      The paper is well written. It uses clear language and provides a
      compact introduction to previous work. To my knowledge, the idea is novel.
      Length scales are a perennial problem in kernel regression models, so the
      promise of a lightweight, efficient and effective approximate
      probabilistic solution (as opposed to the existing alternative of MCMC
      sampling) is a significant contribution. My main concern is the
      experimental evaluation. The experiments presented in Figure 2 do a good
      job of arguing that the new method improves on SPGP-DR. What they don't
      show is whether the new method is actually good at capturing posterior
      mass in the length scales. The paper's point would be strengthened by a
      comparison between the independent marginals (Eq. 11) inferred by the new
      method and those found by a "gold standard", e.g. from an MCMC sampler.
      The variational bound contains some strong factorization constraints (Eq.
      11), and such are known to cause overconfidence. This may, or may not be a
      problem here. I would be grateful if the authors could clarify the
      following points in the feedback: * I understand, from line 293
      that the "full GP" is type-II maximum likelihood, with the ARD kernel? I'm
      surprised that this model is consistently better than the probabilistic,
      Mahalonobis SPGP-DR on the puma dataset. Section 3.3 does not seem to
      explain this. * lines 380 and lines 404-409 argue that the new
      method is fast. But none of the plots quote runtimes. Of course, runtime
      comparisons must be taken with a grain of salt, but they are still
      interesting. Could you quote a few numbers? How fast is VDMGP compared to
      the full GP and to SPGP-DR, for small and large datasets? Finally,
      a minor point about the conclusion: You are suggesting, for future work,
      to use locally varying length scales in the kernel. This is not part of
      the paper so not a core point of my review, but as far as I understand,
      the generalization you propose does not actually give a valid covariance
      function. See Mark Gibbs' PhD thesis (Cambridge), page 18, and Section
      3.10.3 (page 46 and following), which also gives ideas for alternate
      approaches.


Learning to Pass Expectation Propagation Messages
Expectation Propagation (EP) is a popular approximate posterior inference algorithm that often provides a fast and accurate alternative to sampling-based methods. However, while the EP framework in theory allows for complex nonGaussian factors, there is still a significant practical barrier to using them within EP, because doing so requires the implementation of message update operators, which can be difficult and require hand-crafted approximations. In this work, we study the question of whether it is possible to automatically derive fast and accurate EP updates by learning a discriminative model (e.g., a neural network or random forest) to map EP message inputs to EP message outputs. We address the practical concerns that arise in the process, and we provide empirical analysis on several challenging and diverse factors, indicating that there is a space of factors where this approach appears promising.

This paper proposes to learning expectation
      propagation (EP) message update operators from data that would enable fast
      and efficient approximate inference in situations where computing these
      operators is otherwise intractable. - The authors address an
      interesting and relevant problem to the NIPS community and propose -- as
      far as I know -- a novel method of addressing intractable message update
      computations. The approach is reasonably straightforward: to use a more
      computationally demanding approximation process (importance-sampling
      driven Monte Carlo estimates) to generate data for use in learning a
      function approximation (such as a neural network) to provide fast and
      accurate approximate messages. - One of the things I appreciated
      about this paper is its tone. The authors take pains to point out this
      approach is not a panacea and describe the circumstances where such a
      strategy would be effective. The experiments are designed to elucidate the
      strengths and weaknesses of the proposed approach and not necessarily
      restricted to trying to sell it. - Regarding the exposition of the
      material: The paper was largely well written and easy to follow. However,
      I do have one major issue with the organization of the paper. Please
      refactor the paper and supplementary material to ensure that the paper
      stands on its own. Eg. remove descriptions of plots that appear in the
      supplementary material from the paper. This is essentially an abuse of the
      page limit on NIPS papers. - Since computation time of the message
      updates is an important motivation of the proposed scheme, a careful and
      quantitative evaluation of it should be in the paper. It should include
      details of training times as well as "test" times. - Another paper
      relevant to the idea of using function approximation to increase the speed
      of inference is: Efficient learning of deep boltzmann machines. R
      Salakhutdinov, H Larochelle AISTATS 2010. One thing that the above
      paper as well as refs [2], [3] and [11] have in common is their
      consideration of the approximation of inference in the context of
      parameter learning. Unless I missed something, the authors are rather
      silent on the issue of how to use their proposed method in the context of
      also learning parameters. Is this method practical in the inner-loop of
      parameter learning? Do the authors envision a kind of warm restart where
      relatively few training cycles would be required between learning steps?
      It seems like the use of computationally intensive methods such as
      importance sampling in the creation of the training data might render this
      method impractical as an inner-loop to parameter learning. - The
      authors mention that one way their approach differs from these methods
      (mentioned above) is that they are exploiting modularity and that the
      learned message update operators could be re-used in other contexts. This
      is a good point that is likely mostly true, but isn't there some concern
      that the distribution over incoming messages could change from context to
      context and that the learned operator might not be optimal for some of
      these contexts? - Figure 2 needs to be much better explained with
      much more supporting text that it current has.


Recurrent networks of coupled Winner-Take-All oscillators for solving constraint satisfaction problems
We present a recurrent neuronal network, modeled as a continuous-time dynamical system, that can solve constraint satisfaction problems. Discrete variables are represented by coupled Winner-Take-All (WTA) networks, and their values are encoded in localized patterns of oscillations that are learned by the recurrent weights in these networks. Constraints over the variables are encoded in the network connectivity. Although there are no sources of noise, the network can escape from local optima in its search for solutions that satisfy all constraints by modifying the effective network connectivity through oscillations. If there is no solution that satisfies all constraints, the network state changes in a seemingly random manner and its trajectory approximates a sampling procedure that selects a variable assignment with a probability that increases with the fraction of constraints satisfied by this assignment. External evidence, or input to the network, can force variables to specific values. When new inputs are applied, the network re-evaluates the entire set of variables in its search for states that satisfy the maximum number of constraints, while being consistent with the external input. Our results demonstrate that the proposed network architecture can perform a deterministic search for the optimal solution to problems with non-convex cost functions. The network is inspired by canonical microcircuit models of the cortex and suggests possible dynamical mechanisms to solve constraint satisfaction problems that can be present in biological networks, or implemented in neuromorphic electronic circuits.

Overview: The paper studies a neuro-dynamic
      system that is able to solve constraint satisfaction problems (CSP).
      The dynamic equations for neural interaction and synaptic
      weight-changes are relatively straight-forward (equations 1 and 2),
      the archictectures of the networks are relatively complex. The
      elementary computational units Are winner-take-all (WTA) type networks
      that oscillate. It is stated that a crucial property of the elementary
      circuits is their sensitivity to input only at specific points of
      their oscillatory period(s). If the connectivity between such WTA
      units is chosen to reflect a CSP, it is stated that the dynamics
      finally settles into a state that represents a solution to the CSP.
      Simulations of the neuro-dynamics for small and intermediately large
      CSPs empirically show that the neuro-dynamics can find solutions, and
      that it can find approximately optimal solutions if there are no fully
      consistent variable assignments. Evaluation: The
      paper is generally well written and the main points become clear. The
      question what computations neural circuits perform and how they are
      performed is certainly important and interesting. There is also
      evidence for WTA mechanisms that implement a form of possible
      competition between neural untis. Also the effectiveness of input
      during certain points of the phase is an interesting property because
      a small input differences can be amplified (and response can be
      normalized). Even if weakly coupled to an input, such networks can be
      very sensitive. While the general setup is clear, I was missing a
      more extensive discussion about the parameters of the network. Usually
      such parameters have to be tuned. It does not become clear how
      difficult this is for the suggested system. Also a discussion of
      related neuro-dynamic approaches is missing. Neural dynamics that
      become sensitive to input in a critical period of their oscillation have
      been described before, sometimes in the form of sensitivity to initial
      states. Such networks all have a WTA characteristics and at least some
      should be discussed (see, e.g., work by O'Reilly, 2001; Spratling and
      Johnson, 2002; Lucke, Malsburg, 2004). The difference to the here
      discussed system is the goal: here it is finding solutions to CSPs,
      the other networks address component finding in data. Finding CSP
      solutions is a well accepted problem in computer science but it
      becomes much less clear why biological networks should be optimized
      for such a problem - please discuss. Still it is interesting from a
      technical point, and I agree that implementations of inherently
      parallel neuro-dynamic approaches via analog VLSI are very
      interesting. In this respect please discuss two questions, however:
      (1) isn't the complexity of the problem mapped into the complexity of
      the network architecture? (2) can one be confident that the problem
      solving dynamics do not break down if the problem sizes are scaled up?
      What about the total number of variables? The behavior of the
      network is interesting including the behavior for not fully
      satisfiable problems. The empirical finding of exponential scaling is
      not uninteresting but the sampling analogy is taken too far for my
      taste. The energy function could also be used for the satisfiable
      problem (Sec. 3.1). But if the network remains in a solution there, it
      is not spending time proportional to exp(-cE). Additionally, the
      authors themselves say that the network will probably not explore
      high-energy states. So there is only a coarse analogy here as far as I
      can see. Also the criticism of other approaches is too general. Of
      course there are many approaches on the technical side that have
      similar properties. Also it is not clear how computationally costly
      the mechanisms to avoid local optima really are for the studied
      dynamics - time to find optimal solutions may scale exponentially with
      problem size - or the architecture does. In response to author
      feedback: The feedback has clarified most of my questions. I agree
      that scalability will be exponential in the worst case. What would be
      interesting, though, is how the average time to find a solution scales
      with problem size. This should be a point for the revised discussion.
      I also assume that more care is taken regarding the too broad
      criticism of other approaches, that it is discussed why CSP solving is
      interesting for biological systems and about the analogy to sampling.
      Given these improvements, I remain happy with my initial score.


Multiscale Dictionary Learning for Estimating Conditional Distributions
Nonparametric estimation of the conditional distribution of a response given highdimensional features is a challenging problem. It is important to allow not only the mean but also the variance and shape of the response density to change flexibly with features, which are massive-dimensional. We propose a multiscale dictionary learning model, which expresses the conditional response density as a convex combination of dictionary densities, with the densities used and their weights dependent on the path through a tree decomposition of the feature space. A fast graph partitioning algorithm is applied to obtain the tree decomposition, with Bayesian methods then used to adaptively prune and average over different sub-trees in a soft probabilistic manner. The algorithm scales efficiently to approximately one million features. State of the art predictive performance is demonstrated for toy examples and two neuroscience applications including up to a million features.

Summary: This paper addresses the problem of
      conditional density estimation with a high dimensional input space (p
      >> n), an important problems as most (if not all) current models for
      nonparametric conditional density estimation do not scale to
      high-dimensions. Moreover, datasets with high dimensional inputs but
      relatively small sample sizes are becoming increasingly common. The model
      for the conditional density f(y|x) is defined in three stages. First, a
      tree structure is defined over the input space. Second, given the tree
      structure, C_{j,k}, the k^th partition of the X space at scale j, is
      mapped to a lower dimensional space. Third, a local model for the
      conditional density at each node j,k is selected which only depends on x
      through its lower dimensional representation. In practice the lower
      dimension representation is the indicator function 1(x \in C_{j,k}) and
      the local model is Gaussian, i.e. f_{j,k}(y|x)=N(m_{j,k}, s_{j,k}).
      The inference algorithm is “partially Bayesian”, as an empirical Bayes
      approach is used for the tree structure (which is estimated via the METIS
      algorithm). Then, a Gibbs sampling algorithm is used for posterior
      inference of the weights and parameters (m_{j,k}, s_{j,k}) of each node
      indexed by j,k of the tree. Strengths: The authors develop a
      novel method for high-dimensional conditional density estimation which
      lacks any competitors. Performance is quite impressive when compared
      with LASSO, CART, PC regression (even more so considering the added
      flexibility of the model). Weaknesses: Poor notation and lack
      of clear representation of the model for the conditional density. The
      model is stated very vaguely, but then a more specific version is used.
      The paper would be more readable if the focus was only on the specific
      version of the model used in practiced and included a clear mathematical
      description of the model. In equation (1), your model for the conditional
      density given the tree structure tau, depends on some j,k. In particular,
      f(y|x,tau) is a weighted average over the local conditional density of all
      ancestors of j,k. How is this cut-off j,k selected? In equation (1),
      your notation C_{j',k'} \in A_{j,k} does not match your definition of the
      sets, as C_{j',k'} is a subset of the X space and A_{j,k} is a subset of
      the integers. Throughout the text the conditional distribution
      function is used when conditional density should be, leading to some
      incorrect statements (examples are pg. line 86 and pg. 6 line 322).
      The word approximately is used throughout the text, but it may be more
      sensible to replace approximately with equals and state that these are
      model assumptions. What does f_{y|x} represent? The entire collection
      of conditional densities f_{y|x}=(f_{y|x}(\cdot|x), x\in X) or a specific
      f_{y|x}(y|x)? Same comments for F_{Y|X} pg. 4 line 196, there is a
      missing reference. pg. 4 equation (2), should proportional be replaced
      with equals? And the last pi_{j,k}=1-\sum V_{j',k'} (where the sum is over
      the ancestors of j,k). Otherwise it involves a normalization term which
      may affect the full conditional of the V_{j,k}. pg. 5 line 239, the
      notation k_j(x_i) is used but not introduced. pg. 5 line 242, Is the
      full conditional of V_{j,k} correct? Because the likelihood seems to
      involve a normalization term based on V_{j,k}. The paper by
      Bhattacharya, Page, and Dunson (2013) Density Estimation and
      Classification via Bayesian nonparametric learning of affine subspaces
      could be an important reference/competitor to consider. Quality:
      The work is supported by some impressive examples but the model lacks a
      clear mathematical explanation. Clarity: I would suggest removing
      the vague representation of the model and just focus on the specific model
      used in practice. Notation needs some cleaning. Originality:
      Similar to other regression techniques, the proposed model achieves
      flexibility through a tree structure; however, its originality comes from
      flexibility in modelling the entire conditional density not just the mean
      function, which is achieved by allowing for local conditional densities at
      each node of the tree. For inference in high-dimensions, they must take an
      empirical Bayes approach with regards to the tree structure.
      Significance: The results could potentially be very important as
      the allow for conditional density estimation with high-dimensional inputs
      and experimental results are impressive.


Efficient Optimization for Sparse Gaussian Process Regression
We propose an efficient optimization algorithm for selecting a subset of training data to induce sparsity for Gaussian process regression. The algorithm estimates an inducing set and the hyperparameters using a single objective, either the marginal likelihood or a variational free energy. The space and time complexity are linear in training set size, and the algorithm can be applied to large regression problems on discrete or continuous domains. Empirical evaluation shows state-ofart performance in discrete cases and competitive results in the continuous case.

Update after reading the authors' rebuttal:
      Very good paper. I advise the authors, if this paper gets
      accepted, to update it with the additional information regarding rejection
      rates and experimental details. ===== 1. Summary This
      paper presents an efficient algorithm for optimization of inducing points
      in sparse Gaussian process regression. While popular approaches such as
      the IVM use heuristic functions (instead of the marginal likelihood) to do
      this selection, the proposed method uses the marginal likelihood (or the
      variational free energy) to optimize inducing points and GP
      hyper-parameters jointly. Other approaches that achieve this such as the
      SPGP framework of Snelson or the variational approach by Titisias focus on
      continuous inputs, while the proposed approach is more suitable for
      discrete inputs where continuous optimization of inducing points is not
      feasible. The paper extends the Cholesky with Side Information
      (CSI) method of Bach and Jordan to the Gaussian process regression case.
      In particular, it exploits the observation that it is possible to use the
      partial Cholesky factorization as a representation of the approximating
      covariance in the GP. Efficient learning of inducing points is done by
      updating the corresponding matrices in the factorization and by
      approximating the reduction in cost of the objective function (marginal
      likelihood or free energy). Results are shown on several realistic
      datasets with suitable comparisons to standard baselines such as IVM,
      Random and SPGP showing the benefits of their approach, specially in the
      discrete input case. 2. Quality The paper is
      technically sound as it builds upon previous results shown by Bach and
      Jordan (the CSI method) for kernel methods. The claims are supported by
      the corresponding derivations of the updates and the approximations
      required in order to achieve the claimed computational costs. The
      experimental results also strongly support that the proposed method is
      faster than other methods such as CSI, the variational framework of
      Titsias and more effective than other approaches such as random selection
      and IVM. 3. Clarity The paper is generally well
      written providing the detail of the updates necessary for swapping
      inducing points and the actual optimization algorithm used for the overall
      optimization framework. However, it would be useful to clarify the
      following issues: (a) In section 3.1 there is a small k <= m
      mentioned in the factored representation (see e.g. line 137) that affects
      the partial Cholesky factorization and the subsequent QR factorization.
      However, this k seems to disappear in the following sections. Is this
      parameter relevant? if So, how is it set in the experiments?
      (b) The introduction of the "informative pivots" seems quite
      crucial in the fast approximate reduction. However, this means that if one
      were to use this approximation then the method would be optimizing a
      different objective function. While section 3.4 explains that this fast
      approximation is only used to consider the candidates for inclusion and
      that the actual inclusion does evaluate the true changes in the objective
      function, it is unclear how good this approximation is at selecting good
      candidates. In other words, what rejection rates are usually observed
      during optimization? (c) Figure 1 needs improvement. The axes are
      not square (with different numerical bounds) and the labels are very
      small. (d) More details in the experiments are required. For
      example, lines 293-294 mention that swapping inducing points at every
      epoch is unnecessary. How often are the inducing points updated? what
      about hyper-parameters? (e) It is unclear why IVM is more
      expensive than the proposed method as shown in Figure 3 (a). This is
      probably related to item (d) above. (f) More analysis in the
      experiments is needed. Why SPGP is better in terms of predictive
      likelihood as shown in Figure 5? (g) (Minor): The end of line
      174 may need rephrasing: "same for the updating with j" looks like an
      incomplete sentence. 4. Originality The paper
      addresses the well stablished problem of sparsification of GP regression
      via inducing points. It mainly extends the results of Bach and Jordan to
      the sparse Gaussian process regression case. However, it does differ
      substantially from previous approaches in how it selects these inducing
      points and how they can be learned efficiently (and jointly with
      hyper-parameters) while optimizing a single objective function such as the
      marginal likelihood. 5. Significance The results are
      important in that this may establish a superior baseline in terms of
      inducing selection methods for GP regression. The scalability of such
      methods is, however, still an issue for these types of algorithms to be
      used by practitioners on large datasets.