Error-Minimizing Estimates and Universal Entry-Wise Error Bounds for Low-Rank Matrix Completion
We propose a general framework for reconstructing and denoising single entries of incomplete and noisy entries. We describe: effective algorithms for deciding if and entry can be reconstructed and, if so, for reconstructing and denoising it; and a priori bounds on the error of each entry, individually. In the noiseless case our algorithm is exact. For rank-one matrices, the new algorithm is fast, admits a highly-parallel implementation, and produces an error minimizing estimate that is qualitatively close to our theoretical and the state-of-the-art Nuclear Norm and OptSpace methods.

The paper studies the noisy matrix completion problem,
      focusing on entry-wise error guarantees. Via algebraic connections, they
      derive an algorithm for computing the minimum variance unbiased estimator
      for any missing entry. They also provide an explicit expression for the
      minimum variance, show how these results can be extended to higher rank
      matrices, and show promising comparisons with state-of-the-art matrix
      completion algorithms. The paper is inspired by a recent paper on
      noiseless matrix completion, which shows that the set of entries
      recoverable depends only on the mask and in the rank-one case, this
      dependence is made explicit by the connectivity properties of the
      bipartite graph induced by the mask. In the noiseless case, one can
      recover entries by solving a system of polynomial equations defined by
      paths in the bipartite graph. With the appropriate noise model (they use
      multiplicatively-centered finite variance noise) this algebraic connection
      suggests that a polynomial expression from any path between i,j in the
      mask yields an unbiased estimator of the entry A_{ij}. Minimizing
      the variance involves taking the appropriate linear combinations of the
      polynomials defined by paths between i,j. To do this, the authors first
      give an explicit expression for the variance depending only on the
      coefficients, the graph itself, and the the variance of the entries of the
      matrix (assumed to be known). Minimizing this expression leads to the
      MVUE. For higher rank matrices, the polynomial equations aren't
      given by paths and cycles but by higher-order subgraphs, but the same
      arguments can be applied. The simulations conducted show how the
      variance of individual entries depends on the structure of the mask and
      also compare their estimation procedure with the nuclear norm minimization
      and the OptSpace algorithm. In the latter, this approach is favorable in
      the low noise regime, but their error estimates are useful independently
      of algorithm choice. The paper presents a novel analysis of noisy
      matrix completion and uses nice connections to algebraic combinatorics and
      graph theory to arrive at interesting results about estimation. The paper
      is well written and fairly easy to understand, addresses an interesting
      problem with original techniques. Focusing on the rank-one case limits the
      direct applicability of the work but at the same time it significantly
      aids readability and extensions to higher rank settings seem easy enough
      if one can construct a basis for the relevant "circuits." Another
      potential weakness is the use of multiplicative noise rather than additive
      noise, which is much more standard in the statistics community and the
      existing works on matrix completion. It would be nice to give
      frobenius norm error bounds under random sampling as a function of the
      noise variance. I suspect this is not too challenging in the model
      specified using results from random graph theory but it would be even
      better if it could be done with additive noise, which would enable precise
      comparisons with related matrix completion papers.


Reinforcement Learning in Robust Markov Decision Processes
An important challenge in Markov decision processes is to ensure robustness with respect to unexpected or adversarial system behavior while taking advantage of well-behaving parts of the system. We consider a problem setting where some unknown parts of the state space can have arbitrary transitions while other parts are purely stochastic. We devise an algorithm that is adaptive to potentially adversarial behavior and show that it achieves similar regret bounds as the purely stochastic case.

Additional comments after author feedback: I
      thank the authors for answering my questions. I now understand how the
      proposed algorithm works. I do not have any concerns anymore regarding the
      stochastic check. I see how it may be eventually triggered.
      ================================================ Original
      review: The paper describes a new reinforcement learning technique
      that can handle adversarial scenarios. This is a very important problem
      that has not been investigated very much. The proposed algorithm is to my
      knowledge the first that can learn whether each parameter is adversarial
      or stochastic and infer a robust policy based on that. The paper
      is generally well written however, some important aspects of the proposed
      algorithm are not explained. I also have some concerns about the
      stochastic check. In Figure 3, why is Q the minimum of T-t and a
      usual backup? If the rewards are always greater than 1, then Q will always
      be T-t and the algorithm won't be learning anything... I suspect that
      there is a simple explanation, but I'm not sure... Also, in Figure
      3, how is \hat{P} estimated? I couldn't find any description of how the
      transition distributions are estimated. Is it simply the relative
      frequency counts based on the trajectory observed? I'm also having
      trouble convincing myself that the stochastic check is reasonable. It says
      that if the difference between the expected future rewards at (s,a) when
      computed in two different ways is greater than some threshold, then
      Pr(s'|s,a) must be adversarial. The threshold is an expression that
      depends on n and tau, which means that it grows over time. In fact, at
      some point it will be greater than the difference between Vmax and Vmin
      where Vmax = T*Rmax and Vmin = T*Rmin. This means that if the agent has
      not detected an adversarial behaviour by this number of steps, it will
      never find which parameters are really adversarial. This doesn't seem
      right. What will happen if the adversary behaves stochastically long
      enough to ensure that it is not detected and then reveals its adversarial
      nature after that. Would it manage to inflict arbitrary regret?
      The empirical evaluation consists of a proof of concept since it
      includes only a single toy problem that was handcrafted to demonstrate
      that UCRL2 will fail to converge to a good policy. I encourage the authors
      to include a more thorough empirical evaluation with several problems, a
      variation in the degree of adversarial behaviours and to report the
      running times.


Forgetful Bayes and myopic planning: Human learning and decision-making in a bandit setting
How humans achieve long-term goals in an uncertain environment, via repeated trials and noisy observations, is an important problem in cognitive science. We investigate this behavior in the context of a multi-armed bandit task. We compare human behavior to a variety of models that vary in their representational and computational complexity. Our result shows that subjects' choices, on a trial-totrial basis, are best captured by a "forgetful" Bayesian iterative learning model [21] in combination with a partially myopic decision policy known as Knowledge Gradient [7]. This model accounts for subjects' trial-by-trial choice better than a number of other previously proposed models, including optimal Bayesian learning and risk minimization, ε-greedy and win-stay-lose-shift. It has the added benefit of being closest in performance to the optimal Bayesian model than all the other heuristic models that have the same computational complexity (all are significantly less complex than the optimal model). These results constitute an advancement in the theoretical understanding of how humans negotiate the tension between exploration and exploitation in a noisy, imperfectly known environment.

This paper seeks to understand human behavior in
      multi-armed bandit problems by comparing human performance to the
      predictions of four existing models. After presenting a comparison of the
      models’ predictions to human judgments, the authors conclude that the
      “knowledge gradient” (Gupta & Miescke, 1996; Frazier, Powell, &
      Dayanik, 2008) explains human behavior well if it assumes that reward
      distributions may change over time. The paper also includes an analysis of
      some trial-by-trial trends in the data and model predictions.
      --Quality-- I didn't find any errors or conceptual
      problems in the model evaluation or the experimental design.
      --Clarity-- The writing is quite clear in general, with
      some exceptions: * On line 71 “Conditioned on the previous…” this
      sentence might lead a reader to think authors were claiming that KG is
      optimal in general. * Section 3.3’s introduction made be think that
      the evaluation would be based directly on likelihoods of the human
      judgments under each model, and I had to re-read it after looking at the
      results. --Originality-- To my knowledge, this is the
      first paper to compare multiple models against human behavior in
      multi-armed bandit problems, with the aim understanding the trade-off
      between exploration and exploitation. That said, the experimental design
      and the models -- aside from the DBM/FBM comparison -- are not new, and
      past bandit experiments have considered how humans approach the
      explore/exploit trade-off (e.g., Ejova, Navarro & Perfors, 2009).
      --Significance-- My main worry about this paper is that it
      won't attract a wide audience at NIPS, because the conclusions aren't very
      far-reaching and it doesn't appear to have content that would draw an
      algorithms- or applications-oriented audience. Ultimately, the conclusion
      is that a particular model, KG, explains human behavior better than some
      plausible alternatives, and does better when it incorporates the
      assumption that bandits' rewards might change over time. It doesn't make a
      compelling case that the dynamic belief model is necessary in general, and
      there's past work that could be taken to suggest that people approach
      bandit problems as if expecting a changing world (e.g., Shin & Ariely,
      2004). In a longer version of this paper, I'd be excited to see
      (1) a new model that addresses the mismatch between KG and human
      behavior; (2) a larger-scale evaluation using both 0/1 and continuous
      losses, multiple horizons, and whether or not the rewards are actually
      static; (3) a look at transfer learning and how performance changes
      across problem presentations. --Other notes-- There
      were a few things that I wish had been included in the paper or done a bit
      differently. * It would have been helpful to see a visualization of
      how well the models fit different individuals’ judgments, e.g., with error
      histograms. * I wouldn't have used an ANOVA's failure-to-reject to
      claim there's no difference between parameter estimates across different
      models. If the authors think it’s important, they could summarize the
      inferred parameters for each model. Lastly, I noticed a few
      typographical errors: L92: “…all participants played same, but…”
      L128: “…inset shows example Beta prior…” L351: “…always the class
      for…” L406: “…their ability of predicting…” L407: “In real world…”


Restricting exchangeable nonparametric distributions
Distributions over matrices with exchangeable rows and infinitely many columns are useful in constructing nonparametric latent variable models. However, the distribution implied by such models over the number of features exhibited by each data point may be poorly-suited for many modeling tasks. In this paper, we propose a class of exchangeable nonparametric priors obtained by restricting the domain of existing models. Such models allow us to specify the distribution over the number of features per data point, and can achieve better performance on data sets where the number of features is not well-modeled by the original distribution.

This is an interesting paper describing mechanism to
      introduce constraints of the priors 1) I have two concerns with
      how the discussion is formulated: 1.1) The introduction of the paper
      talks about preserving exchangeability, and a substantial effort is
      devoted to showing that restrictions on the predictive distributions do
      not necessarily lead to exchangeable rows. However, given that the authors
      are working with priors on array-valued processes, they should have framed
      the discussion in terms of separate exchangeability (Aldous, 1981), which
      is what we really desired from the model. I think that all the proofs
      works and the models described are indeed separately exchangeable (rather
      than unrestrictedly exchangeable), but the current presentation is not
      really adequate. 1.2) An implicit assumption is that the
      distribution on $\theta$ admits a density (denoted by $\nu$) with respect
      to some dominating measure. Besides not stating this clearly, the authors
      refer to $\nu$ as a measure in various parts of the manuscript.
      Furthermore, the authors refer to $\theta$ as the "directing measure".
      $\theta$ is a collection of probabilities, but they do not add up to 1, so
      talking about a measure here seems incorrect. I think that precision in
      the use of standard terminology is important. 1.3) In equations
      (3) and (4), I believe that the authors should also multiply the final
      expressions by the appropriate indicator (as the support is now
      constrained) 2) Beyond being a mathematically interesting problem,
      the motivation for introducing the constraints is not very strong. The
      best motivation might come from the discussion on HMM models, but even
      that is relatively week. I guess that the authors could argue that
      obtaining better results in the experiment in Section 5.2 is motivation
      enough, but having some intuition on why the proposed model outperforms
      would have been useful. 3) There is no discussion about how the
      size of the truncation is to be selected. Now that constraints have been
      introduced, standard results will not be applicable (particularly if f(S)
      in equation (5) is heavy tailed). 4) Section 3.2 emphasizes that
      directly restricting the predictive distributions does not lead to an
      exchangeable model. The example is nice, but this is a well known property
      and I am not sure that it needed this much space.


Direct 0-1 Loss Minimization and Margin Maximization with Boosting
We propose a boosting method, DirectBoost, a greedy coordinate descent algorithm that builds an ensemble classifier of weak classifiers through directly minimizing empirical classification error over labeled training examples; once the training classification error is reduced to a local coordinatewise minimum, DirectBoost runs a greedy coordinate ascent algorithm that continuously adds weak classifiers to maximize any targeted arbitrarily defined margins until reaching a local coordinatewise maximum of the margins in a certain sense. Experimental results on a collection of machine-learning benchmark datasets show that DirectBoost gives better results than AdaBoost, LogitBoost, LPBoost with column generation and BrownBoost, and is noise tolerant when it maximizes an n ′ th order bottom sample margin.

This paper presents a boosting method that directly
      minimizes the empirical classification error that is defined based on the
      indicator whether predicted y = observed y, the so-called 0-1 loss. The
      proposed method first runs greedy coordinate descent to search the
      coordinatewise minimum loss, and then runs coordinate ascent to expand the
      margin. The method is interesting and novel, and offers some advantages.
      The following concerns are raised. 1. It seems the authors assume
      a hypothesis space which has a finite number of hypotheses because in the
      algorithms 1 and 2, it loops on all hypotheses h at each iteration. For
      instance, algorithm 1 finds all weak learners that lead to largest
      classification error reduction at each iteration. What if users choose a
      hypothesis space, such as linear functions? 2. At the end of
      Algorithm 1, what is the rationale to update the weight of one weak
      learner that gives the smallest exponential loss? Isn’t it still using a
      convex loss function although in the early part of the algorithm, it uses
      0-1 loss. 3. It also assumes that data is always separable as long
      as the combined strong classifier is strong. What if it is not the case?
      What if the chosen hypothesis space is not complex enough to separate a
      given dataset. The paper states that algorithm 1 reaches a solution of
      coordinatewise minimum. It also says the 0-1 loss reaches 0. What is the
      definition of coordinatewise minimum? Is it actually a global minimum
      because the lowest error would be 0. 4. When the second step
      (margin maximization) starts, it starts from a region that would not get
      the solution out of the 0 loss region, characterized by the value of d. It
      is not clear how exactly this value of d is calculated instead of simply
      saying “determine the lowest sample whose margin is decreasing”.
      5. Given separable cases are assumed, the margin is always
      positive in their formulation. What happens if there are negative margins
      for inseparable cases? The second part of the algorithm would not run.
      6. In the early paragraph of page 2, it says “it might escape the
      region of minimum training error in order to achieve a larger margin”. In
      the later section, the design of algorithm 2 will not allow the margin
      maximization step go beyond zero 0-1 loss region. 7. Assuming a
      weak learner can be obtained in a similar computational cost, DirectBoost
      is certainly more computationally heavy for each iteration than AdaBoost.
      In their experiments, what is the stopping criterion that they used for
      AdaBoost? Why does AdaBoost need so many iterations? 8. It is not
      that clear about the truly convincing advantage of the proposed method
      over regular boosting methods.


Bayesian inference as iterated random functions with applications to sequential inference in graphical models
We propose a general formalism of iterated random functions with semigroup property, under which exact and approximate Bayesian posterior updates can be viewed as specific instances. A convergence theory for iterated random functions is presented. As an application of the general theory we analyze convergence behaviors of exact and approximate message-passing algorithms that arise in a sequential change point detection problem formulated via a latent variable directed graphical model. The sequential inference algorithm and its supporting theory are illustrated by simulated examples.

This paper gives a new perspective on the Bayesian
      posterior updates (BPU) asa system of iterated random functions (IRF)
      [4].By proving a general convergence theorem of IRF with semigroup
      property,the convergence of algorithms for change point detection
      problems [17] are analyzed.The major contributions are- to
      establish a general convergence theory for IRF with semigroup property
      (Theorem 1),- to cast existing (exact) algorithms for the classical
      change point problem (CCPP) and the multiple change point problem
      (MCPP) in the IRF framework, and prove their convergence, based on
      Theorem 1.- to propose a fast approximate algorithm for MCPP and prove
      its convergence,based on Theorem 1.I think this is a nice
      paper with a significant impact.BPU is clearly a Markov chain, and so
      it might be natural that it is analyzed in the IRF
      framework.However, the previous convergence results on IRF are not
      directly applicable to BPU,because the Lipschitz condition is not
      satisfied in BPU.Instead, the authors use semigroup property of
      BPU,and establish another type of convergence theory.I expect that
      the established theory would have wide range of
      applications.Pros:- A new perspective on BPU as a special case
      of IRF with semigroup property (3) is given.- A general convergence
      theory (Theorem 1) for IRF with semigroup property is established,
      which might have wide range of potential applications.- A fast
      algorithm (Section 4.1) for MCPP is proposed with its convergence
      guarantees (Theorem 2).Cons:- The relation between the
      theory (Theorem 2) and the experiment (Figure 1) is not clear (see the
      first one in 'Other comments' below).- No practical application of the
      multiple change point detection is introduced (see the second one in
      'Other comments' below).Quality:The paper is technically
      sound, and the claims are supported by theory.Clarity:The
      paper is clearly written, and well-organized.Originality:The
      perspective on BPU as IRF with semigroup property, and the obtained
      convergence theorem (Theorem 1) are novel.Significance:The
      found relation between BPU and IRF with semigroup property, and the
      established convergence theory could be used by other theoreticians to
      analyze algorithms for other problems.Other comments:- It
      would be nice if Eqs.(19) and (20) are somehow expressed in the posterior
      pathgraphs in Figure 1. This would make it clear that the simulation
      supports Theorem 2.In Section 4.3, it is said that 'in some cases,
      they might deviate for a while, but quickly snap back together once
      the change points have occurred.'Readers cannot see this because the
      change points are not indicated.- It would be nice if any practical
      example of MCCP is introduced(I could not find any also in
      [17]).Especially, I wonder in what applications the rulelambda_e =
      min(lambda_s1, lambda_s2) is appropriate.To me, it seems more natural
      to assume that the distribution of X_echanges both at lambda_s1 and at
      lambda_s2.Moreover, if I understand correctly, Theorem 2 guarantees
      the convergence after all the sensors are broken.I suppose people
      are more interested in the convergence after any of the sensors is
      broken, or the sensor on which a user focuses is broken.- Perhaps,
      some thetas in and around Eq.(4) should not be bold-faced.- In Line
      225, 'probabilities 10 by message-passing' looks corrupted.Is 10 a
      citation?- In the last line in Section 5, parentheses for Eq.(29) are
      missing.


On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation
We investigate the relationship between three fundamental problems in machine learning: binary classification, bipartite ranking, and binary class probability estimation (CPE). It is known that a good binary CPE model can be used to obtain a good binary classification model (by thresholding at 0.5), and also to obtain a good bipartite ranking model (by using the CPE model directly as a ranking model); it is also known that a binary classification model does not necessarily yield a CPE model. However, not much is known about other directions. Formally, these relationships involve regret transfer bounds. In this paper, we introduce the notion of weak regret transfer bounds, where the mapping needed to transform a model from one problem to another depends on the underlying probability distribution (and in practice, must be estimated from data). We then show that, in this weaker sense, a good bipartite ranking model can be used to construct a good classification model (by thresholding at a suitable point), and more surprisingly, also to construct a good binary CPE model (by calibrating the scores of the ranking model).

Summary. In more detail, this paper considers
      BC (binary classification), BBR (binary bipartite ranking -- elements of
      one class should all be ranked ahead of the other class), and CPE (binary
      class probability estimation). As discussed in the short review, it is
      shown that a consistent method for BBR, taken as a black box, can be used
      to produce a consistent method for CPE and BC. The key technical point is
      that the rankings must be "calibrated": a sample must be used to either
      figure out where the positive/negative threshold should lie to solve BC,
      or, in the more refined case of CPE, isotonic regression is used to find
      the univariate monotonic map from ranking scores to probability estimates.
      This paper provides these reductions (first with idealized
      distributional access, and then via finite sample estimates), and closes
      with some experiments which show that if one has an existing BBR model and
      some extra samples, then it is better to train the specified reductions,
      rather than just using the sample to train a CPE or BC model from scratch.
      While the experimentation is not extensive, this is exactly the sort of
      sanity check one would like to see. I will note that the use of
      extra samples to calibrate a reduction of this type appears to be new
      (that is, existing reductions which, say, solve BC and BBR via CPE, do not
      need to use extra samples). Evaulation. While perhaps
      not the world's most glamorous problem, it is foundational, and I am happy
      to see progress. The solutions given above are sketched in the body, and
      it is intuitive that some extra data is needed to adapt the ranking
      values. I like the paper, but see one weakness. The definition of
      reduction ("weak regret transfer bound") makes no restrictions on how the
      new samples are used; in particular, whenever class B of problems is
      learnable, then it is possible to reduce from B to any class A simply by
      ignoring A and using the fresh samples to learn B directly.
      Personally, I would like the authors to add some writing to
      address the following points. To be clear, no request is being made for
      new theorems or extensive extra work; simply I think the writing can both
      clarify the above issue, and also pave the way for future work to deal
      with it rigorously. 1. It should be stressed that reductions
      should not be trivial in the sense above (ignoring the provided ranking),
      and moreover that the two presented in this paper are not trivial in that
      way. Moreover, the theorem statements in this paper directly provide the
      reduction, and thus the reader can verify there is no cheating. I note
      that the simulation section is a comparison to the start-from-scratch
      approach, and thus the authors are aware of this issue, but I think it
      deserves significant mention well before that section. 2. It might
      be nice to explicitly state as an open problem the task of formalizing the
      notion of "no cheating", for instance some sort of information theoretic
      approach that asserts these reductions can use fewer samples than what
      you'd need if provided only fresh examples and no ranking function.
      3. As a step in this direction, it may be worth stressing that the
      sample complexity of the binary classification reduction is very nice,
      only paying for thresholds, and not for the full price of learning within
      some potentially large class. Unfortunately, on the other hand, the
      probability estimation reduction pays the full price of isotron, which
      appears unnecessary since only the link function and not the direction are
      used. Thus, another open problem may be reducing the sample complexity in
      the reduction from probability estimation. One more comment, not
      along the lines of the above "weakness". It would be nice if there were at
      least a heuristic argument that extra examples are necessary (i.e.,
      "strong reductions" are impossible). For classification, I think it
      suffices to fix a ranking and show that, without extra samples, one can
      not decide which of two classifications (via thresholding) are better, and
      thus samples are necessary. For probability estimation, similarly
      considering two different links (from a fixed ranking) could work.
      Detailed/Minor feedback. Introduction. I think the
      intro can more cleanly identify the core strategy of the paper. For
      example, by my reading, the basic outline is (1) these binary problems all
      seem related, but we currently have gaps, then (2) the problem is that BBR
      scores can be stretched out without tweaking the loss, which is not true
      for BC and CPE, (3) so we should use a finite sample / distributional
      access to adapt them, (4) for BC this is just a threshold and for CPE it's
      just the link part of isotonic regression. If the authors like this
      approach, I think honoring it could also hint how to break the long
      paragraph in lines 64-81. Line 25. I think the abstract could be made
      more accessible either by defining "Regret transfer bound" or by replacing
      it with an explanation. Line 29. I am not sure this is surprising.
      Line 43. I suggest also citing Tong Zhang's classification consistency
      paper, which Bartlett/Jordan/McAuliffe cite heavily, and which appears in
      for instance the Boucheron-Bousquet-Lugosi survey as the origination of
      these rigorous convex -> zero/one bounds. Line 72. I think this
      would be a good place to discuss the necessity of fresh samples (as
      discussed above). Line 79. Regarding citations for this covering bound
      (which follows by gridding the domain and range?), there are is probably
      earlier work (by a number of decades) in the analysis literature...?
      Line 89. This is nice and I think it could be stressed. Line 125.
      I guess f should map to \bar R, but isotonic regression doesn't handle
      this as far as I know, so I recommend switching to R. Line 177. I am
      not sure the minimizer exists for arbitrary f. In particular, imagine that
      error decreases as t approaches some t0, but then jumps up at t0. This
      should be possible when the pushforward measure through f has a point mass
      at t0. This sort of thing is abusing the fact that "sign" function decides
      0 is positive. Line 215. Rather than "equal parts", might be good to
      stress you can get away with using far fewer samples for the reduction
      step. Line 219. I think this is a nice point. Line 269. Similarly
      to Line 177, I wonder if this minimizer exists. Also, though I didn't
      check your details carefully, I suppose your PAV procedure may be working
      with Lipschitz functions (or at least continuous), in which case this
      really may not be attained. I realized you have a reference here for the
      minimum being attained, but that is a 57 page paper and I must admit I did
      not check for distributional assumptions (ruling out the sorts of
      non-Lebesgue-continuous measures I mention in Line 177). Line 278. I
      suggest parentheses or something around OP3 so it's clearly referring to
      line 269. Line 297. It might be good to mention isotron earlier, for
      instance in the intro as I dicsussed above. Line 307. As mentioned
      throughout, I think some discussion of the n^{-1/3} is warranted. If you
      believe the bound is not tight, maybe state improvements as an open
      problem. Line 341. Just like Remark 8, remark 19 is very nice (note, I
      think the last sentence went astray?). Line 504. Missing an [h] to the
      left of the leftmost equality. Line 602. I guess this 3 is "definition
      3"; similar issues on Line 615.5.


Optimizing Instructional Policies
Psychologists are interested in developing instructional policies that boost student learning. An instructional policy specifies the manner and content of instruction. For example, in the domain of concept learning, a policy might specify the nature of exemplars chosen over a training sequence. Traditional psychological studies compare several hand-selected policies, e.g., contrasting a policy that selects only difficult-to-classify exemplars with a policy that gradually progresses over the training sequence from easy exemplars to more difficult (known as fading). We propose an alternative to the traditional methodology in which we define a parameterized space of policies and search this space to identify the optimal policy. For example, in concept learning, policies might be described by a fading function that specifies exemplar difficulty over time. We propose an experimental technique for searching policy spaces using Gaussian process surrogate-based optimization and a generative model of student performance. Instead of evaluating a few experimental conditions each with many human subjects, as the traditional methodology does, our technique evaluates many experimental conditions each with a few subjects. Even though individual subjects provide only a noisy estimate of the population mean, the optimization method allows us to determine the shape of the policy space and to identify the global optimum, and is as efficient in its subject budget as a traditional A-B comparison. We evaluate the method via two behavioral studies, and suggest that the method has broad applicability to optimization problems involving humans outside the educational arena.

Paper 1279 – Optimizing Instructional Policies In
      this paper, the authors adapt an optimization technique based on Gaussian
      process regression to select the parameters of experiments or teaching
      regime, which will optimize human performance. They evaluate their method
      in two behavioral experiments (one on the presentation rate of studied
      items and another on the ordering of examples while learning a novel
      concept), demonstrating that it is vastly more efficient than traditional
      methods in psychology for exploring a continuous space of conditions.
      Note: I have revised my score to reflect the author feedback's
      assurance that the starting point of Experiment 1 wasn't the optimum.
      However, I still don't fully understand how what they wrote in the
      feedback connects to what they wrote in the paper. I implore the authors
      to make sure it is clear in the final version of their paper.
      There is a lot to like in this paper. They use modern machine
      learning tools to solve a hot problem in cognitive science, optimal
      experimental/instructional design, and validate their method by
      replicating previous memory and concept learning experiments. Their
      adaptation of a previously developed optimization technique is natural and
      clever, albeit somewhat incremental. Besides for one potentially
      catastrophic issue issue with the first experiment (that I will describe
      below), the experiments do a good job of showcasing their method.
      Additionally, the paper is well written and the framing and discussion is
      very appropriate for a NIPS audience (though I have some concerns about
      the framing). However, there are two serious issues with the paper
      that leave me very concerned. 1. Although Experiment 1 may seem at
      first to provide support for their method (as it shows quick convergence
      to the optimal presentation rate), after a closer reading the starting
      condition for the first subject is basically already at the optimum!
      “Subject 1’s training policy, $x_1,$ was set to the middle of the range.”
      The range of the policy space is [ln(.25), ln(5)]. This would make the
      midpoint 0.1116, which corresponds to a duration of 1.118 seconds. The
      reported optimum is 1.15 sec, which is extremely close to the starting
      value for the first subject (1.118 sec). If this is true (and I hope the
      authors can explain why this is not the case!), the new interpretation
      would at best be that the method does not move away from the optimum when
      it already is at it. This basically would invalidate the results of
      Experiment 1, and make the article difficult to support without its
      removal. 2. Although the paper is framed as being for constructing
      “instructional policies” and I am somewhat convinced that POMDP/RL methods
      are prohibitive to adapt for their situation, I really would have liked to
      have seen their method compared against at least one other computational
      method. It seems like there are several methods from optimal experimental
      design that would not be too difficult to adapt to this setting. In fact,
      I am not completely convinced that optimal experimental design is that
      different from finding optimal instructional policies (where instead the
      criterion is maximizing a function rather than discriminating between
      models viewed as functions). At this point there is a long tradition of
      optimal experimental design techniques that work over continuous spaces,
      some of which have already been applied to cognitive psychology. For
      example, Myung & Pitt (2009) Psych Rev. and other work from their
      group (including their 2009 NIPS paper), Kujala (2011) has a review on
      Bayesian adaptive estimation that discusses psychological applications,
      and Vul et al. (2009) in the journal Seeing and Perceiving (available for
      free from Vul’s website). The paper would have been much stronger if there
      was a direct comparison with at least some of these methods (even if they
      had to be adapted). In fact, although the framing of the paper is about
      instructional policies, the experiments and presented results do not seem
      to quite fit it (in the intelligent tutoring systems sense, which would be
      more of a discrete optimization problem). It seemed more about finding
      parameters/conditions to optimize human performance. However, I agree that
      Experiment 2 does a decent job of fitting the framing of the paper.
      Minor comments: Abstract: “… search this space to identify the
      optimum policy.” -> “… search this space to identify the optimal
      policy.” The last paragraph of the introduction seems out of place.
      Perhaps an outline paragraph instead? Second paragraph of Section 2:
      “The approach we will propose likewise requires specification of a
      constrained policy space, but does not make assumptions about the internal
      state of the learner or the temporal dynamics of learning.” No assumptions
      about the internal state of the learner seems a bit too strong. There are
      at least some minimal assumptions inherent in the generative model of the
      student. I agree they are not nearly as strong as some other methods
      though. Third paragraph of Section 2: I would have liked to see a
      little more detail about why numerical approximation methods that account
      for noisy function evaluations would not work. End of Section 2 -
      Section 2.1: Why outline the entire function if you are only interested in
      the maximum (and do not want any assumptions about the internal state of
      the learner)? Section 2.2: you start to use $f_x$ before it is
      defined. “In this paper, we give subjects questions with two equally
      probable choices…” How do you know the choices are precisely 0.5 in the
      minds of the subjects? Discussion of limitations: what about discrete
      parameter spaces? Section 3: If there are 30/d trials and d is
      continuous, what does a fractional trial mean? Figure 3: Label units
      on x-axis (ms). Experiment 2: Why did you have to discard objects with
      [2.75-3.25] graspability from the set? Figure 4: I did not understand
      this plot (I printed it out on B&W, but even looking at the color, it
      does not really make sense to me). Figure 5a: It could be nice to have
      large arrows across the subplots that provide an interpretation of what a
      small vs. medium vs. large fading/blocking policy means for the ordering
      of stimuli (small fading means close to far or random with respect to the
      boundary?)


Latent Maximum Margin Clustering
We present a maximum margin framework that clusters data using latent variables. Using latent representations enables our framework to model unobserved information embedded in the data. We implement our idea by large margin learning, and develop an alternating descent algorithm to effectively solve the resultant non-convex optimization problem. We instantiate our latent maximum margin clustering framework with tag-based video clustering tasks, where each video is represented by a latent tag model describing the presence or absence of video tags. Experimental results obtained on three standard datasets show that the proposed method outperforms non-latent maximum margin clustering as well as conventional clustering approaches.

Summary: This work proposes an extension to
      the maximum margin clustering (MMC) method that introduces latent
      variables. The motivation for adding latent variables is that they can
      model additional data semantics, resulting in better final clusters. The
      authors introduce a latent MMC (LMMC) objective, state how to optimize it,
      and then apply it to the task of video clustering. For this task, the
      latent variables are tag words, and the affinity of a video for a tag is
      given by a pre-trained binary tag detector. Experiments show that LMMC
      consistently, and sometimes substantially, beats several reasonable
      baselines. Comments: Clarity: The paper is clear and
      well-written. The related work section is extensive and the experimental
      details are abundant. Noticed just a few typos: 1) Line 83: The
      "and combines them" should be "and combine them". 2) Line 187: The
      "Balanced clusters are constrained by the last constraint" should be
      something like "Cluster balance is enforced by the last constraint".
      Originality: Adding latent variables to MMC is relatively novel.
      The changes to the MMC objective and optimization are not too substantial,
      but nonetheless represent something new. Very recent work that somewhat
      overlaps with this work is that of Yang Wang and Liangliang Cao,
      "Discovering Latent Clusters from Geotagged Beach Images" (MMM 2013). As
      this work also deals with adding latent variables, though in a different
      manner, it would be good if the authors could indicate the ways in which
      their LMMC method differs. Significance: The empirical validation
      presented in this work is strong enough to indicate that, at a minimum,
      LMMC is significant for video clustering. Most likely it is also
      applicable to many other clustering tasks, which frequently arise in the
      machine learning community. However, it is hard to be sure from the
      current experiments that the latent variables actually correspond to
      coherent sub-aspects of a category. Quality: This paper is of good
      quality. Its math and experiments seem relatively sound. While the delta
      from MMC is not very large in the statement of the objective and the
      exposition of the optimization, the experiments clearly show that the new
      method represents a step up accuracy-wise. One suggestion --- it would be
      interesting to see some runtime comparisons in Table
      1.


Projected Natural Actor-Critic
Natural actor-critics form a popular class of policy search algorithms for finding locally optimal policies for Markov decision processes. In this paper we address a drawback of natural actor-critics that limits their real-world applicability-their lack of safety guarantees. We present a principled algorithm for performing natural gradient descent over a constrained domain. In the context of reinforcement learning, this allows for natural actor-critic algorithms that are guaranteed to remain within a known safe region of policy space. While deriving our class of constrained natural actor-critic algorithms, which we call Projected Natural ActorCritics (PNACs), we also elucidate the relationship between natural gradient descent and mirror descent.

Summary The paper is about the proposal of a
      class of constrained natural actor critics, where, for safety reasons,
      policy parameters must remain in a subregion. The idea is to apply
      natural actor critic algorithms, that update policy parameters by
      following the estimated direction of the natural policy gradient and,
      whenever the policy parameters get out of the safe region, the parameters
      are projected back to allowed values. The authors show that natural
      gradient ascent is a particular case of mirror ascent, and, being the
      latter a constrained optimization algorithm, the projection can be simply
      (and effectively) obtained by adding constraints to the policy parameters
      values. Besides theoretically proving that the resulting projection is
      compatible with the natural policy gradient, a simple example and two more
      complex case studies have been introduced to evaluate the performance of
      the proposed solution and the negative effects that can derive in critical
      systems when either unconstrained optimization or a wrong projection
      method are used. Quality The paper is technically sound. The
      theoretical claims are correctly proved and the effectiveness of the
      proposed approach is empirically evaluated in medium-complexity domains.
      The authors have made a good work in highlighting the advantages of
      the proposed approach, while they have not mentioned to potential weakness
      and no future research direction has been proposed. For instance, I
      would like to read some discussion about the complexity of computing the
      projection and to see some example where the constraints over the
      parameters are not linear. Clarity The paper is well written.
      The paper is structured into many sections. I suggest to reorganize the
      paper into a smaller number of sections that collects the current sections
      as subsection. For instance, I would group sections 3,4,5, sections
      6,7, sections 8,9, and section 10,11. Furthermore, I would move the
      related work section at the end, before the conclusions. I found
      particularly instructing the example of Section 7, that allows the reader
      to clearly visualize how the proposed approach moves in the parameter
      space and what may happen by following other (naive) approaches. For
      what concerns the reproducibility of experimental results, the example in
      Section 7 can be easily reproduced, while for the two case studies some
      details are missing about the domain. In general, no information have
      been reported about the value of the step size used to update the
      parameters along the gradient direction. Originality As far as
      I know, the proposed approach is novel. In the related work section, the
      authors cite other approaches to problems similar to the one faced in this
      paper. As the authors state in the conclusions, the contribution is a
      "simple" modification to standard natural actor-critic algorithms to
      include projection. Such modification is simple since, by choosing the
      correct proximal function, natural gradient ascent can be realized through
      the constrained optimization carried out by mirror descent.
      Significance The setting faced by this paper is really
      relevant. In the last years, policy search and, in particular, policy
      gradient approaches have been largely used to learn policy for complex
      continuous tasks. On the other hand, perform an unconstrained
      optimization in the policy space may be dangerous. This paper advances
      the state of the art by introducing a principled way to constrain the
      search for one prominent class of policy gradient algorithms: natural
      actor critic algorithms. I think that this line of research is
      interesting for many researchers (especially in the field of robotics) and
      this paper can have a positive impact on future researches.


Marginals-to-Models Reducibility
We consider a number of classical and new computational problems regarding marginal distributions, and inference in models specifying a full joint distribution. We prove general and efficient reductions between a number of these problems, which demonstrate that algorithmic progress in inference automatically yields progress for "pure data" problems. Our main technique involves formulating the problems as linear programs, and proving that the dual separation oracle required by the ellipsoid method is provided by the target problem. This technique may be of independent interest in probabilistic inference.

*** UPDATE following author response *** The
      reviewers feel the paper would be significantly strengthened by augmenting
      the paper with some of the additional motivation given in the author
      response. The particular point that seems most promising to me is:
      3. Our ellipsoid-based approach is novel and, we hope, will find
      further applications in the field. Previous polynomial-time solutions to
      CONSISTENCY generally give a compact (polynomial-size) description of the
      marginal polytope. Our approach dodges this ambitious requirement, in that
      it only needs a polynomial-time separation oracle (which, for this
      problem, turns out to be MAP inference). There are many combinatorial
      optimization problems with no compact LP formulation that admit
      polynomial-time ellipsoid-based algorithms (like nonbipartite matching,
      with its exponentially many odd cycle inequalities). This makes it
      sound like there is a chance of finding new polynomial-time CONSISTENCY
      algorithms for a set of models where MAP is tractable but the marginal
      polytope does not have a compact representation. It would be great if the
      authors could spell this out. --------------- The authors
      demonstrate polynomial-time reducibility from several problems
      relating to consistency of pseudomarginals of a graphical model to the
      problems of inference in the same graphical model. Specifically, their
      main result is to show that testing whether a given vector belongs to
      the marginal polytope of G is polynomial-time reducible to the problem
      of MAP inference in a graphical model over G. The reduction works by
      expressing the consistency problem as an LP with exponentially many
      variables. The dual LP, which has an exponential number of
      constraints, can be solved in polynomial time by the ellipsoid method
      if the separation problem can be solved in polynomial time. The
      required separation oracle is a MAP inference problem over G, which
      shows the desired reduction. They present related results that follow
      similar reasoning, e.g., for the problem of finding the closest point
      in the marginal polytope to a provided vector of pseudomarginals.
      The paper is technically sound, but it seems incomplete---it ends
      abruptly without actually presenting several of the results claimed in
      the introduction. I get the sense the authors ran out of space, time,
      or both. I took a look at the result in the appendix about MAX-ENTROPY
      reducing to GENERALIZED-PARTITION. This looks completely standard to
      me: (1) the dual of the maximum entropy problem (problem D-ME in the
      paper) is that of maximizing likelihood in an exponential family, (2)
      this is a convex problem, (3) it can be solved in polynomial time by
      the ellipsoid algorithm if the log-likelihood function and its
      gradient can be computed in polynomial time, (4) the log-likelihood
      function is a linear function minus the log-partition function, so
      computing it efficiently amounts to computing the log-partition
      function efficiently. I don't see what the new contribution is.
      Overall, the authors could do a much better job situating their
      work in relation to previous work. My initial reaction was to guess
      that these results would follow from existing results about the theory
      of exponential family models (cf. Wainwright and Jordan [2]), which
      already describes very strong connections between marginals and
      models (e.g., a bijection between marginals and models in a minimal
      exponential family). I no longer think the main result (Theorem 3.1)
      follows from that literature, but it would help this paper
      considerably if the authors could point out the connections and
      differences. The related work does a very poor job of this
      currently---it simply points to three standard books/monographs about
      graphical models. There are some specific connections to prior
      work that should be mentioned. In particular, the CONSISTENCY problem
      is exactly that of testing membership in the marginal polytope of G.
      The authors should mention this explicitly, as much is known about the
      marginal polytope, and the membership problem which they study has
      appeared before, and is known to be NP-complete: David Sontag
      and Tommi Jaakkola, On Iteratively Constraining the Marginal Polytope
      for Approximate Inference and MAP, Tech. report, 2007 The
      paper is reasonably clear, but imprecise in a few places, especially
      surrounding the problem definitions. In particular, the definition of
      SMALL-SUPPORT is very informal and probably not correct as stated.
      "Polynomial-size" can only be judged asymptotically; for this case
      when you have a specific input instance in hand, you probably need to
      require that the support size is bounded by a specific function of the
      input size. Also, the MAX-ENTROPY problem as stated seems to want to
      simulteneuously maximize entropy and minimize the L1 distance from \mu
      to the marginal polytope. I doubt this is possible. Perhaps you mean
      to maximize entropy subject to the constraint of \nu being a minimizer
      of the L1 distance? Regarding novelty and significance, there is
      really one primary observation in this paper: that MAP inference is
      the dual separation oracle for the LP formulation of the CONSISTENCY
      problem (Theorem 3.1). As far as I know, this is new. Several other
      results (CLOSEST-CONSISTENCY and SMALL-SUPPORT) are slight variants
      that follow from standard LP tricks. The main result is intriguing,
      but it is hard to see its significance. The reductions don't point to
      new algorithms or complexity results for problems that people care
      about (i.e., CONSISTENCY does not arise in practice, and it does not
      imply new hardness results for MAP inference). The paper would be
      strengthened considerably if the authors could point to some
      application of the results. Minor comments * The
      characterization of Markov networks and Bayesian networks on line 31
      seems like it is equating potentials and CPTs with "local marginals".
      I don't agree with this characterization---potentials are not
      marginals. The marginal inference problem is all about converting from
      the potentials to marginals (e.g. [3]). I suggest clarifying the
      wording here. * Figure 1: the text is very hard to read. * Line
      121: the number of LP variables is *equal* to the number of joint
      settings (which is exponential in the number of random variables)
      * Line 162: This wording/notation suggests that the marginal is the
      scalar \mu_{is}. I suggest refining the notation (e.g. (\mu_{is}){s
      \in [k]}) to be more precise, which will clarify that the marginal is
      a vector. The same comment applies to \mu_{ijst}. * Line 167: "Let
      M_1 and M_2 denote the indices (i,s) and (i,j,s,t)". This is a bit
      vague. Can you clarify that M_1 and M_2 are *sets* of indices. *
      Line 394: Do you mean CLOSEST-CONSISTENCY instead of CONSISTENCY here?
      * Line 401: mathematical programming --> mathematical program


First-Order Decomposition Trees
Lifting attempts to speedup probabilistic inference by exploiting symmetries in the model. Exact lifted inference methods, like their propositional counterparts, work by recursively decomposing the model and the problem. In the propositional case, there exist formal structures, such as decomposition trees (dtrees), that represent such a decomposition and allow us to determine the complexity of inference a priori. However, there is currently no equivalent structure nor analogous complexity results for lifted inference. In this paper, we introduce FO-dtrees, which upgrade propositional dtrees to the first-order level. We show how these trees can characterize a lifted inference solution for a probabilistic logical model (in terms of a sequence of lifted operations), and make a theoretical analysis of the complexity of lifted inference in terms of the novel notion of lifted width for the tree.

The paper presents a lifted generalization of
      decomposition trees used to analyze and effective specify a
      computation tree for the analysis of lifted graphical models.
      The paper is exceptionally well-written -- it uses a clean
      notation and is full of examples to guide the reader's understanding.
      From a quality perspective, this paper is among the top rank of papers
      I've reviewed for NIPS. The idea is also novel and seemingly
      worthy of publication -- first-order decision trees (FODTs) strike me
      as a useful internal data structure for guiding the application of
      lifted inference. Yet, on the other hand, it should be pointed out
      that FODTs don't permit novel lifted computations, they only help one
      organize them and analyze a notion of width which bounds the
      computational complexity of evaluation w.r.t. an FODT. In some
      sense the ability to do this computational analysis would have been
      implicit in any prior implementation that had to recursively break
      down a model and apply operations, namely the cited work of Milch et
      al, but again this paper formalizes the recursive structure and
      analysis in a way that I'm not aware has been done before. It
      seems to me that while newcomers to lifted inference will find this
      paper very informative (although perhaps still requiring a lot of
      background reading), I would say that those who have implemented a
      lifted inference system would not at all be surprised by what is being
      presented here. Yet the latter group consists of a quite small group
      of people. In the end, my decision on this paper comes down to
      a few competing issues: (1) The usefulness of having this
      paper in the literature to present a clean formalism for the
      implementation and analysis of lifted inference. (2) The
      computational analysis would have been implicit in previous approaches
      (that certainly had to use a recursive decomposition of the problem
      and for which computation at each step could be bounded). (3) The
      paper does not enable any novel efficient inference, it simply
      provides a clean way to implement and analyze existing inference
      techniques.


Moment-based Uniform Deviation Bounds for k-means and Friends
Suppose k centers are fit to m points by heuristically minimizing the k-means cost; what is the corresponding fit over the source distribution? This question is resolved here for distributions with p ≥ 4 bounded moments; in particular, the difference between the sample cost and distribution cost decays with m and p as m min{−1/4,−1/2+2/p} . The essential technical contribution is a mechanism to uniformly control deviations in the face of unbounded parameter sets, cost functions, and source distributions. To further demonstrate this mechanism, a soft clustering variant of k-means cost is also considered, namely the log likelihood of a Gaussian mixture, subject to the constraint that all covariance matrices have bounded spectrum. Lastly, a rate with refined constants is provided for k-means instances possessing some cluster structure.

The paper adresses the problem of derivation of
      generalization bounds for the k-means objective. If the distribution is
      bounded, standard techniques provide good generalization bounds. Also,
      good bounds can be obtained under strong tail assumptions (e.g. for
      sub-gaussian distributions). The main focus of the current paper is to
      derive generalization bounds based only on moment assumptions. The
      main result of the paper is that if the distribution has p\ge 4 moments,
      then the convergence rate is upper bounded by m^{min(-1/4,-1/2+2/p)}
      (here, m is the number of examples). Evaluation: -----------
      In the realm of machine learning, boundness is fairly reasonable
      assumption. Nevertheless, the centrality of the k-means objective
      justifies research such as the one reported in this paper.
      Unfortunately, I did not have enough time to read and understand
      the proof, which seems quite involved. Therefore, I avoid evaluating
      technical quality and novelty (however, as written, the results and the
      proofs seems non trivial). Of course, I can't vouch for correctness.
      Last remark - due to the lengthy proof and the centrality of the
      result, I would fill more confident if the paper will undergo a journal
      level refereeing. This is a pity and of course unfair for the authors.
      Nevertheless, in the scientific community, correctness comes before
      fairness.


Learning Prices for Repeated Auctions with Strategic Buyers
Inspired by real-time ad exchanges for online display advertising, we consider the problem of inferring a buyer's value distribution for a good when the buyer is repeatedly interacting with a seller through a posted-price mechanism. We model the buyer as a strategic agent, whose goal is to maximize her long-term surplus, and we are interested in mechanisms that maximize the seller's long-term revenue. We define the natural notion of strategic regret -the lost revenue as measured against a truthful (non-strategic) buyer. We present seller algorithms that are no-(strategic)-regret when the buyer discounts her future surplus -i.e. the buyer prefers showing advertisements to users sooner rather than later. We also give a lower bound on strategic regret that increases as the buyer's discounting weakens and shows, in particular, that any seller algorithm will suffer linear strategic regret if there is no discounting.

This paper explores the problem of online learning
      near-optimal posted-prices when facing the same strategic buyer in
      repeated auctions. Assuming the buyer’s discount factor is large enough,
      when the buyer value is an unknown fixed value, the authors propose an
      algorithm which monotonically decrease posted-prices and show that the
      algorithm only suffers sublinear regrets. When the buyer value is drawn
      from a distribution, the authors propose a multi-phase
      exploration-exploitation algorithm with no-regret. The exploration is not
      just for estimating the buyer value, it also discourages buyers to
      manipulate the algorithm. In the end, the authors also show that if the
      buyer discount factor is not *large enough*, it’s impossible to have any
      no-regret algorithms. In general, I like this paper and think it
      addresses a very interesting problem. Most other online pricing papers
      usually assume buyers are myopic or just participate once. It’s nice to
      see that given some conditions, no-regret algorithms are still possible
      even if the buyer participates multiple times and are strategic.
      Below are some detailed comments. - In the stochastic
      value setting, you assume the set of possible prices is finite. Although I
      think it’s fine to assume finite possible prices (if we consider the
      minimum unit of currency), the result is still slightly unsatisfying
      especially your regret bound depends on both the number of possible prices
      and \Delta_p (\Delta_p would be very small if the prices are close.) That
      being said, I still like the current result. I just think it would be much
      nicer to see discussion for infinite or large number of possible prices.
      - The analysis is sometimes hard to follow. In particular, some
      notations are used without being defined (e.g., c_{beta,gamma} and
      d_{beta,gamma} in Theorem 1.) I know they are defined in the appendix, but
      it would be helpful to mentioned it in the main text. It would also be
      helpful to put more intuitions in the main paper. For example, there are
      some nice proof intuitions in the appendix (e.g., the proof in Appendix
      B). I think it would help to put some of them in the main text if space
      allows.


Learning Efficient Random Maximum A-Posteriori Predictors with Non-Decomposable Loss Functions
In this work we develop efficient methods for learning random MAP predictors for structured label problems. In particular, we construct posterior distributions over perturbations that can be adjusted via stochastic gradient methods. We show that any smooth posterior distribution would suffice to define a smooth PAC-Bayesian risk bound suitable for gradient methods. In addition, we relate the posterior distributions to computational properties of the MAP predictors. We suggest multiplicative posteriors to learn super-modular potential functions that accompany specialized MAP predictors such as graph-cuts. We also describe label-augmented posterior models that can use efficient MAP approximations, such as those arising from linear program relaxations.

The paper adapts a random MAP predictor model for
      structured prediction, and proposes to learn the model by minimizing the
      PAC-bayesian bound. They discussed different possible choices of the
      posterior distributions, including multiplicative models of non-negative
      parameters for sub-modular models, and posterior distributions for
      approximate MAP solutions (but without much details) . Quality:
      median. 1. The paper compares with structured SVM. But more
      natural comparisons may be methods in [6,22] which also learns random MAP
      predictions. It would be nice to provide experimental comparison to these
      methods. 2. It would be nice to present results on more than one
      datasets. Clarity. The paper is well written overall. But some
      part of Section 4 confuses me a bit. In particular, 1). Corollary
      1 assumes additive posterior distributions, but right after it, the
      statement that "every smooth posterior distribution remains quadratic
      form" seems to over-generalize the result to any smooth posterior
      distribution. Since this is considered as one of the important
      contribution to the paper. Please you explain this point more clearly and
      explicitly in the paper. 2). Section 4.1 proposes to build
      posterior distributions directly for approximate MAP predictions. The idea
      is interesting, but it seems that no solid example of this kind is
      outlined, and that is no related experiments provided (am i correct?).
      This makes me feel that this section is more a "future-looking"
      discussion. If this true, please state this fact explicitly in the paper,
      since otherwise it may confuse the readers. Also, is it necessary to state
      Theorem 3 as a theorem? This is really a quite obvious result; stating it
      as a theorem may make readers feel that this is an important important
      contribution of the paper (especially in contrast to the other two
      corollaries you used). Give that no solid result is provided in section
      4.1, I suggest to significantly shorten this section and possibly
      rearrange its position (e.g., putting it after Section 4.2).
      Originality: median or low. The paper learns a random MAP
      predictor model for structured prediction by directly minimizing the
      PAC-bayesian bound. The random MAP framework has been proposed in [6,22,
      8] and the loss minimization method is basically the same as the method in
      [8] (probably the main difference is that [8] restricts on additive and
      Gaussian posterior, but the technical difference seems very minor). I
      think the real contribution of this paper is on proposing to use more
      general posterior distributions, in particular, a multiplicative models of
      non-negative parameters for sub-modular models, and (potentially)
      posterior distributions for approximate MAP solutions. But note that the
      paper only shows results on multiplicative posterior models; although
      section 4.1 talks about building posterior models for approximate MAP, the
      paper does not show any concrete example of this kind.
      Significance. The method proposed in the paper may be useful to
      improve accuracy of structured prediction problems.


Transfer Learning in a Transductive Setting
Category models for objects or activities typically rely on supervised learning requiring sufficiently large training sets. Transferring knowledge from known categories to novel classes with no or only a few labels is far less researched even though it is a common scenario. In this work, we extend transfer learning with semi-supervised learning to exploit unlabeled instances of (novel) categories with no or only a few labeled instances. Our proposed approach Propagated Semantic Transfer combines three techniques. First, we transfer information from known to novel categories by incorporating external knowledge, such as linguistic or expertspecified information, e.g., by a mid-level layer of semantic attributes. Second, we exploit the manifold structure of novel classes. More specifically we adapt a graph-based learning algorithm -so far only used for semi-supervised learningto zero-shot and few-shot learning. Third, we improve the local neighborhood in such graph structures by replacing the raw feature-based representation with a mid-level object-or attribute-based representation. We evaluate our approach on three challenging datasets in two different applications, namely on Animals with Attributes and ImageNet for image classification and on MPII Composites for activity recognition. Our approach consistently outperforms state-of-the-art transfer and semi-supervised approaches on all datasets.

This paper describes how to attack the zero-, one-, or
      few-shot recognition problem, where we have a fair amount of training data
      for some classes, but none or very few for some other classes. It does
      this using three different techniques, all combined in a single framework:
      using semantically-meaningful mid-layer knowledge (attributes), building a
      graph on new classes to exploit the manifold structure, and finally by
      using an attribute-based representation for building the graph structure
      (rather than low-level features), which improves performance. The method
      is evaluated on 3 different datasets (Animals with Attributes, ImageNet,
      and MPII Cooking composites), and shows improved performance on all
      compared to the state-of-the-art (slightly). On the plus side, the
      paper's methods are well-motivated and can fit into the same optimization
      framework, regardless of which kinds of side- or ground truth-annotations
      are available. The experiments appear to be conducted fairly and show
      slight improvements on the existing state-of-the-art (significantly for
      the MPII dataset). The paper is well-written. The main negative is
      simply that it is somewhat surprising that performance is not improved
      more by doing all of this. I also have some minor suggestions to make the
      paper a bit easier to read. - Exploiting the similarities of new
      classes w.r.t. existing classes seems very similar to the "simile
      classifiers" of Kumar et al. in their ICCV 2009 paper "Face verification
      using attribute and simile classifiers." - line 74: rewrite to
      "...models do not have to be retrained..." - Figure 3b: reorder
      legend to put red curves on top, then blue curves, then black -
      Figure 4a: accuracy is misspelled - supplementary material table 2
      caption: underlying is misspelled


A multi-agent control framework for co-adaptation in brain-computer interfaces
In a closed-loop brain-computer interface (BCI), adaptive decoders are used to learn parameters suited to decoding the user's neural response. Feedback to the user provides information which permits the neural tuning to also adapt. We present an approach to model this process of co-adaptation between the encoding model of the neural signal and the decoding algorithm as a multi-agent formulation of the linear quadratic Gaussian (LQG) control problem. In simulation we characterize how decoding performance improves as the neural encoding and adaptive decoder optimize, qualitatively resembling experimentally demonstrated closed-loop improvement. We then propose a novel, modified decoder update rule which is aware of the fact that the encoder is also changing and show it can improve simulated co-adaptation dynamics. Our modeling approach offers promise for gaining insights into co-adaptation as well as improving user learning of BCI control in practical settings.

In brain-computer interface (BCI), co-adaptation is an
      important issue, because not only the decoders, but also the brain
      activities can also adapt through feedback. To investigate this problem,
      the paper introduced a split linear quadratic Gaussian (LQG) system. Based
      on the theoretical framework used in the multi-agent control theory, the
      authors proposed interleaved algorithm combining linear-quadratic
      estimation (LQE) for decoder updates and linear-quadratic regulator (LQR)
      for encoder updates with recursive least squares (RLS) to estimate the
      system parameters of the other agents (Procedure 1). They modified further
      the co-adaptive decoder by incorporating r-step ahead estimators of the
      encoder parameters (Procedure 2). For proof of concepts, a numerical
      experiment with both procedures was conducted and the results looks
      promising. Quality: Although I did not check all the
      equations, the paper is technically sound except for the second line of
      Equation (13). It should be a scaler, but the matrix multiplications do
      not become scalars, probably. The authors employed a simple LQG system to
      investigate the co-adaptation problem. The simulation results nicely
      demonstrated that the procedures work as intended in a rather ideal
      situation. However, it is not clear from the manuscript how useful their
      algorithms are in actual BCI applications. Clarity: The paper
      is well organized and clear enough. Originality: I was not
      aware of a theoretical framework for investigating co-adaptation problem
      in BCI community (I have not read important references). The authors have
      in mind 3D tracking applications of BCI with invasive devices, where the
      LQG system is more appropriate. As future extensions, incorporating some
      nonlinearity is necessary to deal with other type of BCI applications.
      Significance: Although there is no validation of the proposed
      procedures with practical BCI applications, I think this work could
      inspire many BCI researchers who are looking for efficient feedback
      schemes to enhance BCI performances and to increase the number of BCI
      users. Based on the above criteria, I would think that this
      paper is a fair contribution marginally above the threshold. One reason is
      that such a theoretical framework could inspire other researchers in the
      field.


Estimation, Optimization, and Parallelism when Data is Sparse
We study stochastic optimization problems when the data is sparse, which is in a sense dual to current perspectives on high-dimensional statistical learning and optimization. We highlight both the difficulties-in terms of increased sample complexity that sparse data necessitates-and the potential benefits, in terms of allowing parallelism and asynchrony in the design of algorithms. Concretely, we derive matching upper and lower bounds on the minimax rate for optimization and learning with sparse data, and we exhibit algorithms achieving these rates. We also show how leveraging sparsity leads to (still minimax optimal) parallel and asynchronous algorithms, providing experimental evidence complementing our theoretical results on several medium to large-scale learning tasks.

This paper studies the effect of data sparsity on
      learning and optimization. Specifically, the paper assumes that the
      support of gradient of an objective function is in the support of feature
      vectors, where feature vectors are considered to be sparse. Definding
      appearance probabablities on feature, the paper derives a lower bound on
      the gap between an estimated solution from minimization with given data
      and a true minimizer, and shows that ADAGRAD methods [4,8] achieves this
      lower bound (and thereby optimal). The paper also suggests an algorithm
      using asynchronous updates, showing that the suggested algorithm achieves
      the optimal bound up to some factors depending on log(dimensionality) and
      update delay probabilities. Quality: The claims are supported
      by proved theorems. Although the claims are technically sound, I believe
      the main assumption in Eq.(2) excludes many popular approaches with
      regularization, for example L1 (lasso), L2 (ridge), group lasso, etc. The
      weakness is indeed revealed in the experiments, for instance the
      right-hand side plots in Figure 2. The plots shows the sensitivity of the
      suggested algorithm to the choices of the steplength. In the experiments
      the authors chose "optimal" steplengths by using test sets, but this
      doesn't sound very practical. Also, it is assumed that the
      distance from independence is bounded by some constant m, that is,
      di(\rho) <= m, but I believe, depending on systems, it could be
      proportional to the number of parallel workers k. If it is the case, then
      the linear speedup using multiple cores in the end of Section 3.2 is
      unlikely. The machines used in the experiment had only a limited number of
      cores, so the validity of assumption was not very well supported.
      Clarity: In general the arguments are clearly stated and well
      organized. But some minor points can be improved for better readability:
      - page 2, line 104: consist distributions -> consist of
      distributions - page 3, Eq (6) and (7): It would be kind to note the
      fact that R_2 ~ O(\sqrt(d)) for less-trained eyes. - page 3, the last
      line: exponential improvement in the dimension: it is not clear to me why
      "exponential" in the dimension. - page 5, line 236: Assuming the data
      appearance... -> Assuming the feature appearance Originality:
      I believe the introduction of the assumption in Eq.(2), the lower
      bound in Proposition 1, the introduction of feature appearance probability
      in this context, are new. The analysis for Hogwild algorithm seems to be
      new as well, but I didn't go through the proof of that part.
      Significance: The new analysis supporting asynchronous update
      when features are sparse, will be an interesting message to the community.
      However as the assumption in Eq.(2) excludes many interesting problems, I
      don't think it will have a large impact.


Locally Adaptive Bayesian Multivariate Time Series
In modeling multivariate time series, it is important to allow time-varying smoothness in the mean and covariance process. In particular, there may be certain time intervals exhibiting rapid changes and others in which changes are slow. If such locally adaptive smoothness is not accounted for, one can obtain misleading inferences and predictions, with over-smoothing across erratic time intervals and under-smoothing across times exhibiting slow variation. This can lead to miscalibration of predictive intervals, which can be substantially too narrow or wide depending on the time. We propose a continuous multivariate stochastic process for time series having locally varying smoothness in both the mean and covariance matrix. This process is constructed utilizing latent dictionary functions in time, which are given nested Gaussian process priors and linearly related to the observed data through a sparse mapping. Using a differential equation representation, we bypass usual computational bottlenecks in obtaining MCMC and online algorithms for approximate Bayesian inference. The performance is assessed in simulations and illustrated in a financial application.

Summary: The paper proposes a multivariate stochastic
      process for modeling time series which incorporates locally varying
      smoothness in the mean and in the covariance matrix. The process uses
      latent dictionary functions with nested Gaussian process priors; the
      dictionary functions are linearly related to the observations through a
      sparse mapping. The authors outline MCMC and online algorithms for
      approximate Bayesian inference and assess performances using simulation
      and processing of financial data. Quality: The paper extends the
      application of the nested Gaussian process priors in [23] to the
      multivariate case and employs them for both the mean and covariance. This
      constitutes a sensible extension, and the authors develop an effective
      inference algorithm. The authors outline an online algorithm and
      suggest that it could prove beneficial for high-frequency data. While this
      is certainly attractive, the paper lacks any clear characterization of the
      computational overhead (or complexity). Nor is there any indication of the
      reduction in accuracy that is induced by executing the online method as
      opposed to rerunning the full posterior computation. As such it is
      difficult for the reader to gain a sense of the settings in which the
      online approach might prove useful and appropriate. The authors do
      not provide compelling evidence that the extended model proposed in this
      paper is important in a practical setting. The simulation examples are
      very toy cases tailored to the technique, so that one would be
      disappointed if the proposed strategy did not provide improved results.
      The studies certainly hint at a setting where the local smoothness in the
      model could prove beneficial, but they do not model a practical
      measurement setting. The paper provides an analysis of correlation
      estimation for national stock indices, with a qualitative analysis of the
      derived correlation results. Unfortunately, there is limited discussion of
      the choice of hyperparameters for the inference and as a result it is
      difficult to determine whether the comparison between LBCR and BCR in
      Figure 3 is meaningful. The BCR graph has the appearance of a correlation
      estimate that is constructed using much greater smoothing than the LBCR
      method. It is not clear that LBCR is using less smoothing where necessary
      and similar levels of smoothing where appropriate; it just appears to be a
      much less smooth estimate of the correlation throughout the entire
      time-series. I wonder whether BCR might achieve a more similar result with
      different parameter choices. Since there is no ground truth, it is
      impossible to know whether the generally increased level of correlation is
      real or an artifact of the analysis. One cannot know if the oscillations
      in first portion of the data (2004-2006) are a true reflection of
      significant variation in the correlations over this period or if they
      simply indicate that there has been inadequate smoothing in the formation
      of the estimate. Although the authors provide plausible explanations for
      the changes in the LBCR estimates, it is not at all clear that these were
      hypothesized before the results were obtained. If not, then one cannot
      place much value in the explanations, since it is almost always possible
      to find some plausible explanation for an increase or decrease in
      correlation of financial time series at the national level. The
      final section of the paper suggests that the model could be employed to
      obtain better predictions of the log-returns of the national stock
      indices. The results here add very little of value to the paper. The
      authors suggest that the boxplots in Figure 4 indicate improved prediction
      performance, but this figure provides no real evidence of any meaningful
      improvement. It is extremely unlikely that any analysis conducted on only
      7 weeks of log-returns (at the weekly level) would be able to provides any
      evidence of any improvement in estimation quality. Clarity: The
      paper is well-written and easy to follow. At times it is not completely
      clear what choices have been made in algorithmic simulation and data
      analysis comparisons. Originality: The model in the paper is a
      variation of the model in [22], replacing Gaussian process priors with
      nested Gaussian process priors. The use of nested Gaussian process priors
      is suggested in [23], where it is applied to the univariate case, for the
      mean only. The paper is not significantly original. Significance:
      The introduced model is a relatively minor adaptation of an existing model
      and the inference techniques are fairly straightforward adaptations of
      existing methods. Although there is some novelty and the model and method
      are of some interest, the paper is unlikely to have a significant impact.


Prior-free and prior-dependent regret bounds for Thompson Sampling
We consider the stochastic multi-armed bandit problem with a prior distribution on the reward distributions. We are interested in studying prior-free and priordependent regret bounds, very much in the same spirit than the usual distributionfree and distribution-dependent bounds for the non-Bayesian stochastic bandit. We first show that Thompson Sampling attains an optimal prior-free bound in the sense that for any prior distribution its Bayesian regret is bounded from above by 14 √ nK. This result is unimprovable in the sense that there exists a prior distribution such that any algorithm has a Bayesian regret bounded from below by 1 20 √ nK. We also study the case of priors for the setting of Bubeck et al. [2013]   (where the optimal mean is known as well as a lower bound on the smallest gap) and we show that in this case the regret of Thompson Sampling is in fact uniformly bounded over time, thus showing that Thompson Sampling can greatly take advantage of the nice properties of these priors.

Paper summary: -------------- The paper
      provides an analysis Bayesian regret of Thompson sampling (where Bayesian
      regret is defined as an average regret weighted according to a prior
      distribution over the parameters) and regret bounds for Thompson sampling
      in the setting of the paper of Bubeck, Perchet, and Rigollet 2013, where
      it is assumed that the mean reward of the best arm and the gap to the
      second-best arm are known, but the identity of the best arm is unknown. In
      this setting the authors improve the upper bound by a log(log(1/epsilon))
      factor, thus closing the gap with the lower bound down to a constant.
      Quality, clarity, originality, and significance:
      ------------------------------------------------ The paper is very
      technical and impractical. Most of the paper is devoted to technical
      proofs that could be moved into appendices, whereas no motivation
      whatsoever is provided for the model studied. Main criticism:
      --------------- As already said, my main criticism is about lack
      of motivation behind studying the model studied by the authors. In the
      first setting considered the authors assume that a prior over parameters
      (arm biases) is provided and study the performance of the model with
      respect to this prior. It would be good to have some examples, where this
      assumption and evaluation measure are plausible. In the second and third
      settings the authors go into further extreme and assume that the bias of
      the best arm and the gap to the second-best arm are known and only the
      identity of the best arm is unknown. Once again, there are no examples,
      where this assumption is satisfied. There are also no examples or
      discussion of how the results of the paper could be useful for solving
      some other theoretical problems. So we have neither practical nor
      theoretical motivation. From looking at the details of the proofs (which
      are the main part of the paper) it is hard to see whether any of the ideas
      could be reused in some other settings. ========================
      Response to the rebuttal ======================== I thank the
      authors for their response. I would like to point out that providing the
      reader a motivation why to read your paper is not called "waste of time
      and space". Going straight to the formulas could probably be considered
      marginally acceptable at COLT (although I would criticize it there as
      well), but at NIPS you have a much broader audience and not everybody is
      dealing with your specific problem on a daily basis. I would like to point
      out that none of my questions were actually addressed. I asked to give at
      least one specific example (out of "hundreds of papers written since the
      70s"), where evaluation of an algorithm with respect to a known prior over
      the parameters would be a reasonable thing to do, but got none. I asked to
      give some motivation for your study of the BPR setting - you are referring
      me for an example to the review of Reviewer_7, but Reviewer_7 just says
      that it "can be useful in real applications" and provides no example, so I
      cannot consider it as a satisfactory answer to my question. I have read
      the BPR paper and it provides very clear motivation for studying the model
      in the setting they study it; they have very clear proofs with ideas that
      are easy to reuse; and they have a lot of results that go far beyond the
      BPR model, such as lower bounds. But even after reading the BPR paper I
      see no motivation to study the problem in the setting you study it and it
      is your job to explain why is it interesting. And it is not explained on
      page 2 you are referring me to. And one more point - you did not
      discuss how your analysis of Thompson sampling in the Bayesian setting
      relates to Bayes-UCB.