The Total Variation on Hypergraphs -Learning on Hypergraphs Revisited
Hypergraphs allow one to encode higher-order relationships in data and are thus a very flexible modeling tool. Current learning methods are either based on approximations of the hypergraphs via graphs or on tensor methods which are only applicable under special conditions. In this paper, we present a new learning framework on hypergraphs which fully uses the hypergraph structure. The key element is a family of regularization functionals based on the total variation on hypergraphs.

The paper defines a notion of total variation on
      hypergraphs, proves its convexity and relation to cuts, discusses
      applications to learning and normalized cuts (for clustering), computes
      proximity operators as part of optimization algorithms, and presents
      numerical experiments comparing with a popular method of Zhou from 2006.
      Overall, the paper tours several sub-disciplines. There are a lot of ideas
      and a few theorems (though of course they cannot say much about a global
      solution in the case of the normalized cut problem, but no one can). The
      paper stays focused despite the amount of material and it is well written,
      with only occasional grammar issues. Section 5 falls most into my
      area of expertise, and I am satisfied with it. I am less familiar with
      existing work on spectral clustering and existing approaches for
      hypergraphs. This paper is basically claiming to be the first major
      advance in 7 years (since Zhou [11]), and I am inclined to believe this,
      but not completely certain. If it is true, then this paper should
      definitely be accepted and will likely become well-known. But of course I
      am a bit skeptical that the authors have not found any more recent work
      than [11], given the amount of attention to the subject (or is it because
      everyone is working on tensors and ignoring hypergraphs?) Other
      comments: - Comparisons were always with [11], which uses the same
      hypergraph framework, and then reduces it with the CE technique to a graph
      problem. But these problems (SSL, clustering) can be attacked from quite
      different perspectives. A comparison with an "outside" approach would make
      this paper stronger, even a simple algorithm (such as k-means for
      clustering... or explain why it is not applicable). Your clustering
      approach which recursively splits clusters is clearly not optimal (though
      I do not doubt that it is a standard technique). - Line 121: "the
      optimal cut". Shouldn't this be "an optimal cut", since it is symmetric so
      and the mirror-image cut would work as well? - The section around
      Thm 4.1 seemed vague to me and could have benefited from more explanation.
      For example, it's not clear to me how to apply the second equation from
      the theorem. - I'm not sure what the authors mean by "tight
      relaxation" (section 4, and mentioned in the intro). Is it the "the
      tightest"? (in that case, define what is your class of "relaxations",
      since it's not the class of convex functions in this case). Or do you mean
      "tighter" than the linear eigenproblem relaxation? (in this case, how do
      you justify this? For graphs, it is justified by empirical evidence, so
      for hypergraphs, you should derive the linear relaxation and test it).
      - Section 5. The method in [24] has since been extended a lot;
      see, for example, Condat 2011
      (http://www.optimization-online.org/DB_HTML/2011/12/3284.html). In
      particular (and I think this was already in [24]), it can exploit smooth
      functions. Usually, it is better (in terms of convergence rates) to take
      derivatives of smooth functions when possible, as opposed to calculating
      their proximity operators; so for the first G term in Table 1, why not
      treat it as the smooth term? For line 276, "the main idea" (also, note
      that this phrase is repeated three times on this page) is not really
      described that well, since this is not unique to the method of [24].
      Rather [24] allows one to separate many terms. - I am quite
      familiar with TV on a grid (for 2D). In this case, people prefer using the
      isotropic TV, but this does not have a closed-form proximity operator nor
      an efficient algorithm to compute it. Alternatively, the most common form
      of anisotropic TV is much simpler to work with. It appears that your TV
      definition generalizes this anisotropic TV. It's not clear, but it would
      be very interesting to explore, if it is possible to define an isotropic
      TV on a hypergraph. - Prop 5.1. This is O( n log n) due to the
      sort. Is it not possible to avoid the sort and just find the largest and
      smallest entries of the input? This would bring it down to O(n). I realize
      you may have to worry about cases when the answer will have several values
      at the max/min, but it still may be possible to deal with this. -
      Experiments. Overall, well done. For making the hypergraph edges using
      data points that have the same value of a feature, is it not beneficial to
      make edges (but with much lower weight) when data points have *similar*
      values? - The table on page 8: what are the entries? Some kind of
      error? This was not explained - Line 415: "Our method... minimizes
      the normalized cut on the hypergraph." Unless I have misunderstood, this
      is not true. You apply a convex method to a nonconvex problem, so in the
      best case (and this you would need to show) you reach a stationary point
      or local minima. So at least say that your method "attempts" to
      minimize... Grammar: - Line 276 and 301, "proximum" and
      "proxima" are not commonly used; just use "proximity operator". -
      Line 416-7 is awkward. - The authors use the construction "allow
      to" a lot, as well as variants "suggest/recommend to use", "favor to split
      off", "propose to solve", (e.g., line 17, 69, 107, 192, 201, 385). Since
      the English is otherwise nearly perfect, it's worth correcting this. In
      these cases, the infinite should be a gerund, e.g. "suggest to use" should
      be "suggest using". Rarely, both forms are correct (e.g. "I like to play
      tennis" and "I like playing tennis" are both fine). In the case of
      "Hypergraphs allow to encode", it should be "Hypergraphs allow one to
      encode". - "However" is not used correctly a few times. E.g. line
      199, it should be "Hence" not "However", and line 408 it should be
      "Thus".


Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints
We investigate two new optimization problems -minimizing a submodular function subject to a submodular lower bound constraint (submodular cover) and maximizing a submodular function subject to a submodular upper bound constraint (submodular knapsack). We are motivated by a number of real-world applications in machine learning including sensor placement and data subset selection, which require maximizing a certain submodular function (like coverage or diversity) while simultaneously minimizing another (like cooperative cost). These problems are often posed as minimizing the difference between submodular functions [9,25] which is in the worst case inapproximable. We show, however, that by phrasing these problems as constrained optimization, which is more natural for many applications, we achieve a number of bounded approximation guarantees. We also show that both these problems are closely related and an approximation algorithm solving one can be used to obtain an approximation guarantee for the other. We provide hardness results for both problems thus showing that our approximation factors are tight up to log-factors. Finally, we empirically demonstrate the performance and good scalability properties of our algorithms.

The authors introduce two new submodular optimization
      problems and investigate approximation algorithms for them. The
      problems are natural generalizations of many previous problems: there
      is a covering problem (min f(X) st. g(X) >= c) and a packing or
      knapsack problem (max g(X) st. f(X) <= b), where both f and g are
      submodular. These generalize well-known previously studied versions of
      the problems usually assume that f is modular. They show that there is
      an intimate relationship between the two problems: any polynomial-time
      bi-criterion algorithm for one problem implies one for the other
      problem (with similar approximation factors) using a simple reduction.
      They then present a general iterative framework for solving the two
      problems by replacing either f or g by tight upper or lower bounds
      (often modular) at each iteration. These tend to reduce the problem
      at each iteration to a simpler subproblem for which there are existing
      algorithms with approximation guarantees. In many cases, they are able
      to translate these into approximation guarantees for the more general
      problem. Their approximation bounds are curvature-dependent and
      highlight the importance of this quantity on the difficulty of the
      problem. The authors also present a hardness result that matches their
      best approximation guarantees up to log factors, show that a number of
      existing approximation algorithms (e.g. greedy ones) for the simpler
      problem variants can be recovered from their framework by using
      specific modular bounds, and show experimentally that the simpler
      algorithm variants may perform as well as the ones with better
      approximation guarantees in practice. Quality: This is an
      extremely high quality paper that makes numerous contributions. I
      especially like the fact that it presents a unifying framework from
      which many specific results (new and old) can be derived. Thus, it
      contributes not only to the compendium of approximation results for
      specific submodular optimization problems, but also to the general way
      of thinking about the research area. The insights relating
      approximability to curvature seem to be a separate and interesting
      thread of contribution. It is a very complete piece of work.
      Clarity: The paper is also impeccably written. It packs in many
      results and is thus very dense. But it is well organized and follows a
      very logical outline that first presents the main ideas and then uses
      them to instantiate a number of specific results. My one suggestion to
      the authors is that, for the short version of the paper, perhaps you
      can consider omitting some of the lesser results to include a bit more
      discussion. For example, the extensions in Section 4.3 could easily
      be relegated to the longer version. Two things that would add to the
      paper are: (1) some interpretation of approximability results and (2)
      a bit more explanation of the experiments. For example, I had
      the following question: it seems that the ellipsoidal-based algorithms
      (e.g. EASSC) give the strongest approximation-guarantees. How should I
      think about the relationship between the guarantee given by ISSC and
      EASSC? In practice is the EASSC bound usually better?
      Originality: while there has been a significant amount of work on
      submodular optimization, and this paper touches on many specific
      pieces of prior work, it seems to be quite original in the way it
      relates all of the prior work within a unifying framework.
      Significance: I am not an expert in this area, but I think it
      could be highly significant. One question is how often the most
      general version of the problem (submodular cost + submodular covering
      constraint) really crops up in practice vs. the previously studied
      special cases. Minor comments: * Line 157: "A simple
      modification of the iterative procedure, however, guarantees
      improvement". This seems like a counterpoint to some previous point,
      but I can't figure out which. Perhaps it was orphaned. * Typo in
      Corollary 4.10: \sqrt{n} \log --> \sqrt{n} \log n * The experiment
      setup is a bit hard to follow. It would be nice to be able to
      understand this without referring to prior work. Can you elaborate a
      bit on the application? It seems like you are trying to select
      utterances which correspond to a small set of words, while trying to
      capture the most variety in the utterances. Is this right? What is the
      relevance of this problem?


Local Privacy and Minimax Bounds: Sharp Rates for Probability Estimation
We provide a detailed study of the estimation of probability distributionsdiscrete and continuous-in a stringent setting in which data is kept private even from the statistician. We give sharp minimax rates of convergence for estimation in these locally private settings, exhibiting fundamental trade-offs between privacy and convergence rate, as well as providing tools to allow movement along the privacy-statistical efficiency continuum. One of the consequences of our results is that Warner's classical work on randomized response is an optimal way to perform survey sampling while maintaining privacy of the respondents.

This paper studies minimax bounds for probability
      estimation under the constraint that the estimation must preserve privacy
      of the individual data. The authors consider a privacy definition called
      local privacy. They study two probability estimation problems, multinomial
      estimation and density estimation. In both problems, the authors show
      sharp minimax rates of convergence. They demonstrate that, for the
      discrete multinomial estimation problem, local privacy causes a reduction
      in the effective sample size quadratic in the privacy parameter alpha.
      Since alpha can often be seen as a small constant, the effective sample
      size is of the same order as the non-private case. For the density
      estimation problem, the authors demonstrate that the optimal rate for the
      non-private setting is no longer attainable if local privacy must be
      preserved. Overall the results are very interesting. As far as I
      could read, the proofs are correct. To the best of my knowledge, the
      minimax bound for density estimation under privacy constraint has not been
      considered before. My main comment is that the minimax bound of
      the multinomial estimation is closely related to previous works on the
      noise complexity for differential privacy, but there is a lack of mention.
      In particular, two papers consider highly relevant problems,
      Hardt&Talwar, On the geometry of differential privacy, STOC, 2010; and
      De, Lower bounds in differential privacy, TCC, 2012. These two papers
      study worst case lower bounds for the error of linear queries under the
      constraint of differential privacy. The probability estimation problem
      considered in this paper is actually a special case of the linear counting
      query studied in those two papers. Also, the measures used for the error
      are the same L_2 metric. The only difference is that this paper considers
      local differential privacy while Hardt&Talwar and De consider
      differential privacy. Local privacy posts stronger constraint than
      differential privacy, and therefore lower bounds for differential privacy
      are also lower bounds for local privacy. I am wondering if the bound for
      local privacy given in this paper improves over previous bounds for
      differential privacy. For the density estimation problem, the
      statement of the result is a bit confusing. The authors state that the
      lower bound in the local privacy setting is higher than the non-private
      setting. But the minimax bound in this paper is for the special case that
      the density can be expanded with trigonometric basis. I am wondering if
      the lower bound for the non-private setting eq.(13) holds for the general
      Sobolev space as given in definition 1 or for the special case of
      trigonometric basis. It is a fair comparison only if the non-private lower
      bound holds for the trigonometric basis. Additional comments to
      the rebuttal: The feedback partially clarifies the relation to
      previous worst case lower bounds. But note the work of Nikolov, Talwar,
      and Zhang, The Geometry of Differential Privacy: The Sparse and
      Approximate Cases (STOC 2013) also considers Mean Square Error. I suggest
      the authors add their explanations and missing references to the paper.


Near-optimal Anomaly Detection in Graphs using Lovász Extended Scan Statistic
The detection of anomalous activity in graphs is a statistical problem that arises in many applications, such as network surveillance, disease outbreak detection, and activity monitoring in social networks. Beyond its wide applicability, graph structured anomaly detection serves as a case study in the difficulty of balancing computational complexity with statistical power. In this work, we develop from first principles the generalized likelihood ratio test for determining if there is a well connected region of activation over the vertices in the graph in Gaussian noise. Because this test is computationally infeasible, we provide a relaxation, called the Lovász extended scan statistic (LESS) that uses submodularity to approximate the intractable generalized likelihood ratio. We demonstrate a connection between LESS and maximum a-posteriori inference in Markov random fields, which provides us with a poly-time algorithm for LESS. Using electrical network theory, we are able to control type 1 error for LESS and prove conditions under which LESS is risk consistent. Finally, we consider specific graph models, the torus, knearest neighbor graphs, and ǫ-random graphs. We show that on these graphs our results provide near-optimal performance by matching our results to known lower bounds.

This paper presents a new algorithm to detect
      activation in a weighted graph, as well as some theoretical analysis
      regarding minimal detection levels of this algorithm as well as
      others. The problem considered - detecting activation in graphs is
      (relatively) new and interesting. Furthermore, the new solution
      proposed by the authors is to the best of my knowledge highly
      original, and may be of applicability in other problems. That
      said, the paper is not very clearly written, and is not easy to read and
      follow. Detailed comments are as follows: p.1 L36 - Detection
      is broader than as written in first sentence of intro. Consider for
      example change detection, outlier detection etc. In many such problems the
      null is not merely noise. p.2 L57-72: Not clear why authors
      spend these lines presenting these applications. On first read I
      thought they actually had data from these applications and would show
      the results of their algorithms. The wording on line 55 is quite
      misleading "we will examine two real-world examples". In fact, can the
      authors point to links where such data is available in the public
      domain ? p.2 L85: is C required to be a single connected component
      ? Does not seem so from definition, and constraint that out(C) \leq
      rho means that for sufficiently large rho C can be a collection of a
      quite large number of singletons. p2 L87 number of edges -->
      total weight ? p.2 L100 typically one cannot control both types of
      error, but rather tries to maximize probability of detection while
      fixing the false alarm rate. p2 L106 "instead of the test being
      randomized" not clear. p3 top - it is not clear that as p\to\infty
      there is a 0/1 limiting law of perfect distinguishability or
      indistinguishability (type I plus type II errors tending to 0 or to 1).
      p3 L135 "used to localize binary signals over graphs" not clear.
      p4 L165 "that do not take the graph into account" is not clear.
      p4 L172-177 - whole paragraph is not very clear. Also, w.r.t.
      proposition 2 and the problem formulation, I would like to point out
      that there may be a difference between detection of presence of
      activation in the graph, and estimation of which nodes where activated.
      See for example work by Y. Ingster (1997) and by Donoho and Jin,
      Annals of Statistics, 2004. I presume there may be a similar phenomena
      here, unless there is a constraint that C is connected ? p.4
      L177 tends to provide worse performance - is this an empirical observation
      or is there a mathematically precise statement. p 5 can the
      authors comment on the complexity of the LESS algorithm as a function of p
      rho etc. p.7 how does r_C depend on rho ?


q-OCSVM: A q-Quantile Estimator for High-Dimensional Distributions
In this paper we introduce a novel method that can efficiently estimate a family of hierarchical dense sets in high-dimensional distributions. Our method can be regarded as a natural extension of the one-class SVM (OCSVM) algorithm that finds multiple parallel separating hyperplanes in a reproducing kernel Hilbert space. We call our method q-OCSVM, as it can be used to estimate q quantiles of a highdimensional distribution. For this purpose, we introduce a new global convex optimization program that finds all estimated sets at once and show that it can be solved efficiently. We prove the correctness of our method and present empirical results that demonstrate its superiority over existing methods.

The one-class SVM (OCSVM) algorithm, due to by
      Scholkopf et al, finds a hyperplane in feature space that maximizes the
      separation between the training set and the origin. The original OCSVM
      paper showed that, for suitable choices of kernel and slack variable
      penalty, the function returned by OCSVM is an indictor of the q-support*
      of the data distribution, i.e., a set of minimum volume that contains a q
      fraction of the distribution. A good way to characterize a distribution is
      to estimate its q-support for many values of q, and one could simply run
      OCSVM many times to generate these estimates -- this paper calls that
      algorithm I-OCSVM. However, these estimates won't necessarily be nested,
      while the actual q-supports must be. This paper describes an extension of
      the OCSVM algorithm, called q-OCSVM, for computing nested estimates of the
      q-supports of a distribution. Other papers have also proposed extensions
      of OCSVM with this property, but this paper's approach for enforcing the
      nesting constraint is very simple: the learned hyperplanes are required to
      be parallel, i.e., they only differ in their bias term. The paper modifies
      the OCSVM optimization problem to include this parallel constraint, and
      shows that the dual is still solvable in polynomial time for very
      high-dimensional feature spaces, via the kernel trick. The theoretical
      results in Section 4, which prove that the q-OCSVM algorithm
      asymptotically estimates the q-supports, are straightforward extensions of
      the results found in the original OCSVM paper. The experimental results
      show that the q-OCSVM algorithm outperforms existing algorithms, including
      I-OCSVM, for several datasets, though sometimes the improvement is small.
      Some might regard the simplicity of the q-OCSVM extension as a
      reason not to accept the paper, but I view it as a point in the paper's
      favor. However, I do wonder whether an even simpler approach might work
      too. For example, I accept that forcing all the hyperplanes to be parallel
      acts as a kind of regularizer, and thus helps with overfitting. But
      couldn't this effect be achieved with the I-OCSVM algorithm by just
      increasing the penalty on the norm of the weight vector? Admittedly, this
      still would not guarantee that the estimated q-supports are nested. Which
      leads to a question: Does I-OCSVM intersect the q-supports that it learns,
      as suggested in the Introduction? Also, is q-OCSVM faster to train than
      I-OCSVM? It seems like it might be, since it's one QP and not several QPs.
      If so, that would be worth mentioning. Finally, instead of setting \nu = 1
      - \alpha, what if we used a held-out validation set to choose a value for
      \nu that optimizes coverage ratio? It seems that might improve the results
      for all algorithms. The results in Figures 2 and 3 are averaged
      over 61 and 21 datasets, respectively, but the plots do not contain error
      bars, so it's hard to judge the significance of the results. On the other
      hand, the fact that the q-OCSVM algorithm dominates the others for all
      values of q is encouraging. Also, I think the left- and right-hand side of
      each figure contain the same information, and thus seem redundant.
      The paper is clear and well-written, and the relationship to
      previous work is well-explained. Figure 1 is a very helpful illustration.
      For completeness, it might be good to cite some work on estimating the
      quantiles of a _conditional_ distribution. Search for "nonparametric
      quantile regression" or "kernel quantile regression". * The term
      "q-support" is mine, not the paper's.


Stochastic Convex Optimization with Multiple Objectives
In this paper, we are interested in the development of efficient algorithms for convex optimization problems in the simultaneous presence of multiple objectives and stochasticity in the first-order information. We cast the stochastic multiple objective optimization problem into a constrained optimization problem by choosing one function as the objective and try to bound other objectives by appropriate thresholds. We first examine a two stages exploration-exploitation based algorithm which first approximates the stochastic objectives by sampling and then solves a constrained stochastic optimization problem by projected gradient method. This method attains a suboptimal convergence rate even under strong assumption on the objectives. Our second approach is an efficient primal-dual stochastic algorithm. It leverages on the theory of Lagrangian method in constrained optimization and attains the optimal convergence rate of O(1/ √ T ) in high probability for general Lipschitz continuous objectives.

This paper presents an algorithm for multiple
      objectives stochastic convex optimization. The issue with multiple
      objectives is handled by choosing one objective to minimize and by
      constraining all the others via lower bounds. The stochastic sampling
      is addressed by averaging the costs on the available samples. The
      proposed algorithm is then a primal-dual projected gradient descent.
      The authors show also that 2 naive implementations yield a worse
      solution error bound than the primal-dual approach. Quality
      -- The paper seems technically sound. I have checked several steps
      but not all. The main serious shortcoming is that there are no
      experiments at all to support any of the claims made in the paper.
      Overall, this seems an unfinished paper. Although this is mostly a
      theoretical paper, I'd recommend to add experiments to show advantages
      and limitations of the proposed strategy (transfer m objectives to
      constraints). For example, how does one choose the bounds for each
      objective function? What is the best way to pick the objective to be
      minimized? Can one draw any connection of Pareto optimal solutions?
      Clarity -- Overall the paper is clear. The authors
      should also show some illustrations together with the theory.
      Typos/unclear line 183: what is \hat f_i ? line 215: to fail
      for -> fail for Originality -- The originality of
      this work is difficult to assess. It seems that Algorithm 1 and the
      formal proofs are new, but I am not an expert in the field and
      relevant literature could have escaped me. The idea of moving all but
      one objectives to the set of constraints seems quite simple and
      with several limitations. Significance -- The use of
      multiple objectives as well as stochastic sampling is present in a few
      applications (some shown at the end of the Introduction) and worthy of
      attention. However, the resulting algorithm is quite a straightforward
      idea, and has some limitations (optimization is with respect to only
      one objective function) that are not fully examined in the paper.
      The full significance of the algorithm is not demonstrated with
      experiments either. The main contribution then is left to the
      proof of the error bounds (Theorems 1 and 2).


Matrix Completion From any Given Set of Observations
In the matrix completion problem the aim is to recover an unknown real matrix from a subset of its entries. This problem comes up in many application areas, and has received a great deal of attention in the context of the netflix prize. A central approach to this problem is to output a matrix of lowest possible complexity (e.g. rank or trace norm) that agrees with the partially specified matrix. The performance of this approach under the assumption that the revealed entries are sampled randomly has received considerable attention (e.g. [1,2,3,4,5,6,7,8]). In practice, often the set of revealed entries is not chosen at random and these results do not apply. We are therefore left with no guarantees on the performance of the algorithm we are using. We present a means to obtain performance guarantees with respect to any set of initial observations. The first step remains the same: find a matrix of lowest possible complexity that agrees with the partially specified matrix. We give a new way to interpret the output of this algorithm by next finding a probability distribution over the non-revealed entries with respect to which a bound on the generalization error can be proven. The more complex the set of revealed entries according to a certain measure, the better the bound on the generalization error.

Existing bounds for matrix completion from the trace
      and \gamma_2 norms require that the observed sample positions be chosen
      either randomly or from a specific deterministic strategy. This paper, in
      contrast, considers the matrix completion problem when the set of observed
      samples is arbitrary. A bound is derived on the error of the reconstructed
      unobserved entries in terms of a probability distribution over those
      entries which must satisfy certain conditions, and the error bound
      improves with the complexity of the observed sample set. The probability
      distribution involved in the bound can be optimized in order to provide
      the tightest reconstruction guarantee. Quality: This paper
      considers an important problem and derives a novel bound that applies to
      matrix reconstruction using any norm that satisfies a certain property
      (including the trace and \ell_2 norms). The bounds in this paper can be
      somewhat difficult to interpret at first glance, but the author(s) do a
      satisfactory job in helping the reader interpret the results. The fact
      that the probability distribution involved in the bound can be optimized
      is a plus. Clarity: This paper is clearly written, and the proof
      of the main result is surprisingly concise. Originality: Existing
      bounds for matrix completion from the trace and \gamma_2 norms require
      that the observed sample positions be chosen either randomly or from a
      specific deterministic strategy. This paper is original in allowing the
      sample positions to be chosen arbitrarily. Significance: Extending
      the literature on matrix completion guarantees to allow for arbitrary
      sample positions is a significant contribution. The bounds themselves in
      this paper can be somewhat difficult to interpret, and it is conceivable
      that other researchers may derive bounds of different flavors in the
      future. However, this paper seems like a strong first step in a new
      direction. Some minor comments: Line 045: Figure 3 should be
      Figure 1? Line 070-071: missing a minus sign in exponent? Line 100: do the
      author(s) really do anything to "enhance" Alg1 in this paper? Line 185:
      the result depends on \gamma_2(Y). Could the author(s) mention whether it
      is possible to relate this to \gamma_2(X)? Line 195: should \lambda be
      \Lambda? Lines 211-213 and 235-238: sometimes symbols are lowercase,
      sometimes uppercase. Is this intentional? UPDATE: Thanks to the
      authors for their notes in the rebuttal
      period.


k-Prototype Learning for 3D Rigid Structures
In this paper, we study the following new variant of prototype learning, called k-prototype learning problem for 3D rigid structures: Given a set of 3D rigid structures, find a set of k rigid structures so that each of them is a prototype for a cluster of the given rigid structures and the total cost (or dissimilarity) is minimized. Prototype learning is a core problem in machine learning and has a wide range of applications in many areas. Existing results on this problem have mainly focused on the graph domain. In this paper, we present the first algorithm for learning multiple prototypes from 3D rigid structures. Our result is based on a number of new insights to rigid structures alignment, clustering, and prototype reconstruction, and is practically efficient with quality guarantee. We validate our approach using two type of data sets, random data and biological data of chromosome territories. Experiments suggest that our approach can effectively learn prototypes in both types of data.

This paper introduces an algorithm to cluster shapes
      into k groups, each containing rigidly transformed versions of one shape,
      and computing a prototype for each group. Two new chromatic clustering
      algorithms are proposed to compute the prototype within a cluster with
      various quality guarantees. I can see the interest of the proposed
      algorithm with its theoretical guarantees. However, from a practical
      standpoint, I am not entirely convinced of the benefits of the resulting
      method. In particular: - Step 2(a) of the 1 prototype learning
      algorithm finds the best alignment of each shape to the current prototype.
      Would is really make a significant difference to simply compute the median
      of the n points corresponding to point q_j, instead of using the chromatic
      clustering algorithm? Since the alignment already guarantees a one-to-one
      correspondence, is it really useful to recompute it with the chromatic
      clustering algorithm? - It is mentioned that ICP would not be suitable
      because it does not guarantee a one-to-one correspondence. This is true,
      but I am not sure how often this problem would occur in practice.
      Furthermore, since the chromatic clustering algorithm recomputes the
      correspondences, it may not be relevant at all. - In practice, how
      important is it to run several iterations of Step 2(a) and (b)? I would
      expect that, if the clusters found by [11] are good, a single iteration is
      sufficient. If the clusters are bad, then it would be hard to find a
      meaningful prototype, anyway. - While the experiments show the good
      behavior of the algorithm, they do not compare it against other existing
      methods (e.g., an ICP-based approach, or simpler versions of the algorithm
      as proposed above). It is thus not clear what the true benefits of the
      algorithm are. It also seems to me that several references to
      existing shape analysis works are missing. In particular: - The
      manifold of 3D shapes has been studied by Kendall, and later by Dryden
      & Mardia. - A lot of work on the topic has been recently done by
      Srivastava and colleagues, e.g., Kurtek et al. CVPR'10. - Other shape
      representations have also been studied, such as level sets (Osher &
      Fedkiw), or medial surfaces (Bouix et al.). Additional comments:
      - The multi-level net method seems related to a branch-and-bound
      approach. Would it be possible to get better guarantees with a proper
      branch-and-bound approach? For instance, Hartley & Kahl, IJCV'09,
      studied branch-and-bound to optimize rotations, although in a different
      context. - In Definitions 5 and 6, why change the notation from P_i to
      G_i? - Line 233, it should be m median, instead of k. - In the
      synthetic experiments, were the rigid shapes transformed by random rigid
      transformations? With the current description, it seems that they just are
      noisy versions of the prototypes, which would make the task easier.


A Scalable Approach to Probabilistic Latent Space Inference of Large-Scale Networks
We propose a scalable approach for making inference about latent spaces of large networks. With a succinct representation of networks as a bag of triangular motifs, a parsimonious statistical model, and an efficient stochastic variational inference algorithm, we are able to analyze real networks with over a million vertices and hundreds of latent roles on a single machine in a matter of hours, a setting that is out of reach for many existing methods. When compared to the state-of-the-art probabilistic approaches, our method is several orders of magnitude faster, with competitive or improved accuracy for latent space recovery and link prediction.

This paper proposes parsimonious triangular model
      (PTM), which constrains the O(K^3) parameter space of mixed-membership
      triangular model (MMTM) to O(K) for faster inference. Authors develop a
      stochastic variational inference algorithm for PTM and additional
      approximation tricks to make it further scalable. It is shown from
      synthetic dataset that the reduction of the number of variables may lead
      to stronger statistical power, and from real-world datasets that the
      proposed method is competitive with existing methods in terms of accuracy.
      Quality: PTM seems to be an interesting specialization of
      MMTM, but it is questionable what is the practical advantage of achieving
      good scalability in terms of K (the number of possible roles). To
      empirically evaluate the value of such a method, it is critical for us to
      answer "how does it help if we can learn MMTM with large K?" Since MMSB
      and MMTM are mixed-membership models, using small K may not be as
      troublesome as it is in single-membership models! For real network
      experiments, what would be really interesting is to see the performance of
      both MMTM and PTM as a function of K: if we can really see that PTM with
      large K outperforms MMTM with small K, it will show the power of PTM.
      Unfortunately such experiments were not done. Actually I could not find
      good description about how K's were chosen in real data experiments for
      both PTM and its competitor MMSB; the method of model selection could've
      biased experimental results. Authors are also a bit reticent on
      what are the impact of simplifying assumptions they make. It is clear that
      we can achieve good scalability by constraining the parameter space, but
      it would be also nice to hear about when would such model fail. If there
      really are O(K^3) distinct values in the parameter matrix B (following
      authors' notation) of MMSB and MMTM, PTM should perform poorly, right? In
      this regard, it is unsatisfactory that the choice of true B in synthetic
      data experiments were not clearly described; authors' should've
      experimented with various B's which some are in favor of PTM and some are
      against the model. It would be more beneficial for readers to know when
      would PTM succeed and when would it fail, than to know only when it would
      succeed. On the other hand, authors introduce a number of tricks
      to make PTM scale; while I find them to be very interesting and effective,
      this paper does a poor job in describing and analyzing the impact of such
      an approximation. 1) What is the impact of choosing the threshold \delta?
      On real-world and synthetic data experiments, does it help to choose
      larger or smaller \delta? How much can we lower \delta to improve
      scalability while not sacrificing much of accuracy? 2) Authors claim that
      a-MMSB does not work since downsampling compromises accuracy; why would
      PTM free of such problem? MMTM and PTM are actually ignoring triplets with
      only one edge. Why can't we ignore non-edges in MMSB while we can in PTM?
      3) What is the impact of O(K) approximation to local update (in page 5)?
      What happens if we sample 2K, 4K, or 100K of triples? Would the quality of
      approximation suffer when K gets larger? To sum up, I am not
      convinced experimental setup was fair to competitors, and the description
      of potential weakness of the algorithm/model is poor. Clarity: the
      paper is very clearly written and well-organized. Originality: In
      terms of modeling it is a bit incremental, but achieving scalability via
      constraining parameters is an interesting and original direction to
      pursue, as recently in this topic a lot of work has been focused on
      generalizing existing models. Significance: Provided simplifying
      assumptions in the model and the inference algorithm do hold in practice,
      it would have a good impact on the practice of latent-space models on
      networks, as not many of them actually scale to large data.
      Additional Comments: Personally I think the title is too broad.
      There are other scalable approaches to other latent space models, and
      authors propose a scalable algorithm to _a_ latent space model- PTM.
      Also, I am not sure that PTM is really a probabilistic method.
      Authors argue in the last paragraph of Section 3 that this is same in
      spirit to bag-of-words assumption, but I am not convinced. When LDA
      generates a bag-of-words, there always exists a document corresponding to
      it; so it has proper likelihood function, no matter how unrealistically
      simple it might be. On the other hand, with very high probability there
      does not exist any network corresponding to a bag-of-triples generated by
      MMTM and PTM. Therefore MMTM/PTM is not really a generative model for
      networks, and majority of posterior probability mass is placed on modeling
      non-networks! It might be dangerous to abandon basic principles of
      likelihood in "probabilistic" methods (what does probability mean in
      PTM?)... I am curious what authors and other reviewers
      think.


Eluder Dimension and the Sample Complexity of Optimistic Exploration
This paper considers the sample complexity of the multi-armed bandit with dependencies among the arms. Some of the most successful algorithms for this problem use the principle of optimism in the face of uncertainty to guide exploration. The clearest example of this is the class of upper confidence bound (UCB) algorithms, but recent work has shown that a simple posterior sampling algorithm, sometimes called Thompson sampling, can be analyzed in the same manner as optimistic approaches. In this paper, we develop a regret bound that holds for both classes of algorithms. This bound applies broadly and can be specialized to many model classes. It depends on a new notion we refer to as the eluder dimension, which measures the degree of dependence among action rewards. Compared to UCB algorithm regret bounds for specific model classes, our general bound matches the best available for linear models and is stronger than the best available for generalized linear models.

Summary: ******* The paper considers the
      following general sequential learning problem: a set of actions \cal{A}
      and a parametric set F of bounded functions mapping actions to the reals
      are given. At each time step, a subset \cal{A}_t \subset \cal{A} is
      provided, one has to choose one action A_t in \cal{A}_t, and then gets a
      reward R_t. The reward is assumed to come from a sub-Gaussian
      distribution with mean f_\theta(A_t), where f_\theta \in F is a fixed,
      unknown, function. The authors then introduce in Section 4 the
      notion of eluder-dimension that is designed to capture how difficult it is
      to infer the value of one action from previously observed ones. A
      general regret decomposition is proposed in Section 5, and a specific
      confidence bounds around a least-squares estimate of f_\theta are
      proposed in section 6. The main result of the paper bounds the regret
      of a UCB-like strategy (Proposition 4) and a Thompson-like strategy
      (Proposition 5) in terms of the eluder-dimension of the hypothesis class
      F, in a distribution-free way. The eluder-dimension is then bounded in
      some specific examples including the case of linear and generalized linear
      bandits. Details: ******* Since the set of
      available actions change over time, the initially proposed setting
      corresponds to a sleeping bandit problem. However it is unclear to me
      what are the assumptions you consider on the sets \cal{A}_t (Standard
      assumptions are: deterministic, stochastic, oblivious, adaptive). It
      also seems to me that from section 6, they are actually assumed to be just
      the set \cal{A}, and especially in section 8 where the main result is
      clearly stated for a "not-sleeping" bandit case ( \cal{A}_t = \cal{A}
      for all t). This is actually confusing and it seems actually unclear
      that the main results hold as well for the general sleeping bandit setting
      you consider in the introduction (especially for an oblivious or adaptive
      choice of \cal{A}_t). Thus I suggest for clarity that you align the
      proposed setting with the main result, that is either you remove the
      sleeping-bandit assumption entirely, or extend your results and compare
      them to the standard sleeping-bandit literature as well. I have the
      feeling the only purpose of this general sleeping-bandit setting is to be
      able to handle the contextual bandit problem as a special case.
      Question: what about distribution-dependent bounds scaling
      with O(\log(T))? Example 2: If the noiseless binary
      classification setting as detailed line 202-205 indeed provides
      interesting motivation for introduction of the eluder-dimension, I
      find on the other hand lines 205 to 212 confusing: Both the VC-dimension
      and the eluder-dimension are quantities that depend on a set of functions,
      not on a specific algorithm. Thus I am not sure to see the goal of those
      lines. Please clarify. Proposition 3: It may be useful to
      emphasize a little more the result of this proposition, since this is
      really the key point that enables you to derive the main regret bounds.
      Other point: In algorithm 1, you use no regularization parameter
      (that is you use \Phi_t and not for instance \Phi_t + \lambda I).
      However in the proof (Section D.2) you introduce such lambda, for an
      upper bound on the max value. It may be useful to clarify if this is
      only needed in the proof or if the algorithm also needs it. You may also
      comment on the absence of lambda w.r.t such algorithms as OFUL. Also the
      assumption that the bound S seems needed but is not reported. Please
      clarify all these points. Minor comments/typos: Line 224:
      missing "of. Line 228: mention that the label of a is defined to be
      f_\theta(a). Line 302: I find the notation using E_t a bit awkward:
      what is E_t? Lemma 1: It may be useful to recall that eta is the
      constant from the sub-Gaussian assumptions. Line 518: "The implies".
      Line 539: Missing “|” before \cal{H}_0 Line 682. Maybe add
      something like "by definition of dim_E(F,\epsilon)", or "otherwise, it
      would contradict the definition of d". Line 699: 1 should be 1/T.
      Line 728: "(f_{\rho_1} - f_{\rho_1})(a_k)" change second \rho_1.
      Line 735: idem. Line 706-707: To be able to deduce that, you
      should replace the previous arguments with w_{i_t} \geq \epsilon (instead
      of >) from line 698. Quality. Good, sound. Clarity.
      Good, very well written. Originality. Good. Significance.
      Good.


Tracking Time-varying Graphical Structure
Structure learning algorithms for graphical models have focused almost exclusively on stable environments in which the underlying generative process does not change; that is, they assume that the generating model is globally stationary. In real-world environments, however, such changes often occur without warning or signal. Real-world data often come from generating models that are only locally stationary. In this paper, we present LoSST, a novel, heuristic structure learning algorithm that tracks changes in graphical model structure or parameters in a dynamic, real-time manner. We show by simulation that the algorithm performs comparably to batch-mode learning when the generating graphical structure is globally stationary, and significantly better when it is only locally stationary.

Paper Summary: The paper tackles an
      interesting and under explored problem of tracking changes in the
      underlying dependency structure between variables in a stream (e.g. time
      based) of data points. The author focus on a efficient
      (computational/memory wise) solution for the simpler SEM models. They
      develop a new algorithm for this task, named LoSST. LoSST uses a standard
      batch learning algorithm for the underlying structure learning (PC by
      Spirtes et al, 2000) and tracks several statistics from the stream of data
      to decide whether to initiate new structure learning. The tracked
      statistics involve the distance from the weighted mean of each variable,
      the correlation matrix (eq. 1,2) and the weighted sample size. For the
      sample weight there is a weighting scheme for each new point based on how
      "surprising" the new data point is (Eq. 3,4). Each point's Mahalanobis
      distance (MD) is computed and the estimated p-values for the MDs by a
      Hotelling's T2 distribution is combined using Liptak's weighted pooling of
      p-values to compute an overall "significance" statistic \rho. This stat is
      then used in a noisy OR configuration with the previous step to decide
      wether to start a new structure learning. The authors compare to vanilla
      PC batch learning on synthetic data sets where either a change in params
      or a change of structure is introduced after a fixed set of datapoints,
      testing for different number of variables and max degree configurations.
      Finally, they perform a short analysis of real life data from US price
      index with 529 data points (every month, starting at 1967) and six
      variables. They show the algorithm detects some known changes/trends in
      the US economy while ignore short lived changes. Pros: 1. The
      paper tackles an interesting and under explored problem. As the author
      point out there is a lot of room for additional development of practical
      solutions for these and other models, as well as matching theory and
      theoretical guarantees. 2. The algorithm is novel and
      practical/efficient. It makes creative use of a host of different methods
      for this. 3. Experiments seem adequate trying to track the effect of
      the various params and include an interesting real life data set. 4.
      Authors give a good review about previous work on this problem. 5.
      Overall well written. Cons: 1. The biggest lack in the
      paper with respect to this interesting problem is that of theory or any
      guarantees under some assumptions. Convergence and diligence are like
      opposing forces as the authors point out (p. 4) but it seems like some
      form of guarantees under simplifying conditions should be plausible. At
      least mention as another direction for future work. 2. Similarly, the
      solution suggested by the authors, LoSST, is heavy on the heuristic/hack
      side. That's OK and enables the authors to get a practical solution, but
      the decisions involved should be more readably noted. Since tracking the
      mean and correlation matrix is straightforward, these involve mostly
      decisions on reweighing, effective sample size, and when/how to re learn.
      The authors make creative use of some methods under conditions that
      probably void guarantees of these methods. For example, they estimate the
      MD p-value using Hotelling's T2 distribution. They set the sample size N
      to be the effective sample size S^r. However, that distribution was
      developed for complete counts from a given distribution, not for
      partial/re weighted counts. If they actually sampled from these
      nets/structures with these partial weights would the distribution follow
      this form? Similarly, Liptak's weighted p-values was developed for meta
      analysis using independent tests with different sample sizes. At least if
      the methods are used outside their guarantees/context the authors should
      clearly state that, so readers don't get the wrong
      impression/understanding. Other comments: - The authors
      make creative use of footnotes to outline proofs, probably beyond what
      Leslie Lamport or the conference organizers intended. Consider using supp
      for that and other missing details. - Figures are way too small, with
      unnecessary space between them, missing labels, missing details in the
      captions, legends etc. Description of the experiments is not clear enough
      (e.g. top paragraph p. 7 is cryptic). - The explanation why LoSST does
      worse on edge addition when only the params are changed is not
      satisfactory. - With regards to the above, it seems desirable that
      sparking params update and structure updates should be decoupled.


Adaptive Submodular Maximization in Bandit Setting
Maximization of submodular functions has wide applications in machine learning and artificial intelligence. Adaptive submodular maximization has been traditionally studied under the assumption that the model of the world, the expected gain of choosing an item given previously selected items and their states, is known. In this paper, we study the setting where the expected gain is initially unknown, and it is learned by interacting repeatedly with the optimized function. We propose an efficient algorithm for solving our problem and prove that its expected cumulative regret increases logarithmically with time. Our regret bound captures the inherent property of submodular maximization, earlier mistakes are more costly than later ones. We refer to our approach as Optimistic Adaptive Submodular Maximization (OASM) because it trades off exploration and exploitation based on the optimism in the face of uncertainty principle. We evaluate our method on a preference elicitation problem and show that non-trivial K-step policies can be learned from just a few hundred interactions with the problem.

This paper extends the concept of adaptive
      submodularity (Golovin & Krause, JAIR 2011) to the setting where
      one wants to maximize an adaptive submodular function whose form is not
      known, but where the necessary training episodes are provided.
      What differentiates adaptive submodularity for submodularity is
      that the set-valued function being optimized is stochastic: the reward
      for selecting set A \subset L can vary with the state/context \phi \in
      {0,1}^L. We assume that \phi is drawn from a distribution P(\Phi). Call
      the reward f(A,\phi). A training episode is a sequence of steps, where the
      forecaster selects a set of items according to a policy. The policy
      accounts for the uncertainty in \phi. The key ideas in this paper,
      which connect adaptive submodularity to multi-armed bandits are, in my
      opinion K1. Learn the incremental gains, not the submodular
      function The critical assumption made is that the reward of adding
      an element to a set does not depend on the unknown state \phi, so one
      can generalize the reward gained from adding an element to a set across
      states \phi. K2. Optimism in the face of uncertainty Once
      you assume the incremental gains do not depend on the context, learning
      can be framed as a bandit problem where each element in the ground set
      corresponds to an arm. Under K1, the authors can frame learning in terms
      of the UCB1 algorithm on Bernoulli bandits, drawing heavily on the
      original UCB1 paper. It is worth mentioning to the reader that the
      no-regret result in this paper depends heavily on K1 and K2.If one looks
      at the literature on learning monotone submodular functions in a more
      general setting (Balcan & Harvey, STOC 2011), the picture is much more
      dismal. The proof of Theorem 1 is quite elegant, but the paper is
      definitely hampered by the tight page limitations of a NIPS submission. A
      reader who has not read Golovin & Krause will have only the foggiest
      idea of why this problem is interesting. One way of improving the clarity
      of this submission, in its own right, is to strip the redundant first
      paragraph in the conclusions, and use the space for exposition on how
      submodularity fits into this problem from a MAB perspective. The
      paper clearly builds on two established ideas: adaptive submodularity and
      the UCB1 algorithm for multi-armed bandits. The clever bit is K1, which
      connects the two ideas. The experiments (Section 4) are
      perfunctory, and seem pretty contrived. The ground set is the 19 genres.
      It's not at all clear what the training episodes are. It does seem though
      that the independence of gains and contexts is the same as assuming that a
      genres of a movie don't depend on the preferred genres of the users. But,
      I'd expect that there are more movies in more popular genres. Is there
      something that I am missing?


Modeling Clutter Perception using Parametric Proto-object Partitioning
Visual clutter, the perception of an image as being crowded and disordered, affects aspects of our lives ranging from object detection to aesthetics, yet relatively little effort has been made to model this important and ubiquitous percept. Our approach models clutter as the number of proto-objects segmented from an image, with proto-objects defined as groupings of superpixels that are similar in intensity, color, and gradient orientation features. We introduce a novel parametric method of clustering superpixels by modeling mixture of Weibulls on Earth Mover's Distance statistics, then taking the normalized number of proto-objects following partitioning as our estimate of clutter perception. We validated this model using a new 90-image dataset of real world scenes rank ordered by human raters for clutter, and showed that our method not only predicted clutter extremely well (Spearman's ρ = 0.8038, p < 0.001), but also outperformed all existing clutter perception models and even a behavioral object segmentation ground truth. We conclude that the number of proto-objects in an image affects clutter perception more than the number of objects or features.

UPDATED AFTER AUTHOR FEEDBACK AND OTHER REVIEWS:
      Based on the author rebuttal and the comments of the other
      reviewers, I still believe the paper is worthy of acceptance. However, I
      agree with the other reviewers that the paper merits a ``1'' (incremental)
      rather than a ``2'' (major novelty) in the impact score. ORIGINAL
      REVIEW: Summary: This paper proposes an image-based model for
      visual clutter perception (``a crowded, disorderly state''). For a given
      image, the model begins by applying an existing superpixel clustering then
      computing the intensity, colour and orientation histograms of pixels
      within each superpixel. Boundaries between adjacent superpixels are then
      retained or merged to create ``proto-objects''. The novel merging
      algorithm acts on the Earth Movers Distance (EMD), a measure of the
      similarity between two histograms. The distribution of histogram distances
      in each image for each image feature is modeled as a mixture of two
      Weibull distributions. The crossover point between the two distributions
      (or a fixed cumulative percentile if a single distribution is preferred by
      model selection) is used as the threshold point for merging: an edge is
      labelled ``similar'', and the superpixels merged, if the pair of
      superpixels exceed the threshold point for all three features. The clutter
      value for each image is the ratio of the final number of proto-objects to
      the initial number of superpixels (i.e. 0 = no proto-objects, not
      cluttered; 1 = all superpixels are proto-objects). The model is
      validated by comparing to human clutter rankings of a subset of an
      existing image database. Human observers rank images from least to most
      cluttered, then the median ranking for each image is used as the ground
      truth for clutter perception. The new model correlates more highly with
      human rankings of clutter than a number of previous clutter perception and
      image segmentation models (including human object segmentation from a
      previous study). Strengths: In my opinion, scene clutter (a
      ``crowded, disorderly state'') is not the most interesting quality of
      perception, but a workable definition of ``set size'' in complex scenes is
      a very important problem. I think this paper is the precursor to an
      important contribution: namely, whether this clutter metric relates to
      performance in behavioural tasks such as search. The method for clustering
      seems to do a good job and to my knowledge is a sufficiently different
      approach to previous methods as to offer a novel contribution. The
      behavioral ranking task is neat and the dataset will be of interest to the
      community for future model testing. The model correlates admirably with
      clutter rankings. I believe the paper to be a good contribution to the
      NIPS community. Weaknesses: It is not clear from the
      manuscript how strongly the success of the clutter metric depends on the
      type and parameters of the superpixel pre-processing. I assume this is
      done to make computation more tractable, but more detail should be
      provided in the manuscript as to why this does / does not matter.
      The paper makes the theoretical (based on previous work) and
      empirical (based on three histograms) claim that the similarity distances
      follow Weibull distributions. I am not convinced beyond doubt that this is
      necessarily true. For example, I wonder whether the histograms in Figure 2
      would be equally well fit by other mixture models using e.g. Gamma
      distributions. However, I don't think this distinction is important for
      the applicability of the method. Clarity: The manuscript is
      generally clearly written, but I believe some re-ordering of
      methodological details could save the reader some effort. For example, we
      are told in Section 2.1 that superpixels are extracted, but not until line
      336 is the actual method referenced. Line 149: ``However, there is no
      evidence showing that similarity distances follow symmetrical or normal
      distributions. Therefore, we argue that the similarity distances xf
      computed by EMD follow a Weibull distribution, which is a distribution of
      the Exponential family that is skewed in shape.'' The ``therefore''
      implies that the Weibull distribution is a logical given, but it's not at
      all clear from this how a Weibull distribution was selected. The choice is
      justified more in subsequent paragraphs but the paper would be more clear
      if these paragraphs were rearranged: first tell us why you selected
      Weibull then tell us ``for these reasons we argue... Weibull''.
      Minor points / suggestions: -- for the initial MLE guess (line
      190): I understand that this would be set separately for every image. Is
      this correct? -- Line 269: ``making p and epsilon the only model
      parameters in our framework'': Should this read ``tau'', not ``p''? If
      not, what does ``p'' refer to here? -- is it correct that each
      image will contain a potentially different threshold point, depending on
      the fitting of the mixture model to the distance histograms from that
      image? -- section 4.1 is unclear to me: does ``image features''
      here refer to the similarity distance metrics in each of intensity, color
      and orientation, or the intensity, color and orientation histograms
      themselves? In the latter case, is it right to say that a circular
      statistic such as orientation is ``upper bounded''? -- lines
      347--350: ``visual clutter perception may ignore feature dissimilarity on
      the order of 14% when deciding whether two adjacent regions are similar
      and should be merged.'' Given that the largest value of the epsilon
      parameter tested was 15%, I would want to see some dramatic drop of
      correlation within that 14--15% range to believe this more. A plot of the
      average correlations against epsilon (with five curves, one for each tau)
      would be informative here. -- it is interesting that human object
      count is such a poor predictor of clutter. Perhaps the authors could
      provide example images from the database to show where clutter rankings
      and object counts diverge. -- is it meaningful to ask why the
      correlations of the clutter model with the median ranking is higher than
      the inter-observer agreement? On line 309 you present the average
      correlation between pairs of participants. What is the correlation of each
      participant with the average of all other participants? That is, does the
      clutter model predict the average ranking better than any individual
      human? -- the reader is often provided with an average rank order
      correlation and a p-value (presumably telling us that this correlation is
      significantly different to zero). I would find it more informative to be
      given the mean correlation and also the range of observed correlations.
      -- First sentence of section 2.4 has an incorrect figure
      reference; should be Fig 2d? -- Line 399 missing capital letter to
      start sentence. -- Line 401 "our work suggests a mean" should be
      "means"


Adaptive Market Making via Online Learning
We consider the design of strategies for market making in an exchange. A market maker generally seeks to profit from the difference between the buy and sell price of an asset, yet the market maker also takes exposure risk in the event of large price movements. Profit guarantees for market making strategies have typically required certain stochastic assumptions on the price fluctuations of the asset in question; for example, assuming a model in which the price process is mean reverting. We propose a class of "spread-based" market making strategies whose performance can be controlled even under worst-case (adversarial) settings. We prove structural properties of these strategies which allows us to design a master algorithm which obtains low regret relative to the best such strategy in hindsight. We run a set of experiments showing favorable performance on recent real-world stock price data.

The paper presents a nice algorithm for trading a
      commodity (e.g. stock of certain type) for a market maker (i.e. an entity
      that must buy from/sell to everybody). The paper nicely explains a
      (somewhat simplified) mechanics of the stock exchange (limit orders and
      market orders). The paper presents a natural set of base
      strategies. The set of base strategies is parametrized by a parameter "b"
      -- the spread between buy and sell price. On top of these strategies they
      authors apply standard online algorithms (Randomized Weighted Majority
      Algorithm, Follow the Perturbed Leader) for hedging/switching between the
      base strategies. What makes the application non-trivial is that any
      strategy (base or not) is stateful -- namely it maintains "cash" and
      "inventory" (i.e. a number of underlying stocks). Since standard
      algorithms "for learning with expert advice" are stateless, this makes
      their application somewhat non-trivial. To overcome this authors prove
      that certain interesting structural properties of set of base strategies
      (e.g. the buy-sell price interval of a base strategy with smaller spread
      is contained in the interval of a strategy with bigger spread). This
      allows the authors to derive a bound on the regret (i.e. convergence)
      relative to the best base strategy in hindsight. The paper is very
      clearly written. I like it very much and I am happy to recommend it for
      acceptance. I have to admit that I don't have time to go through
      the proofs in detail, but all lemmas seem plausible. Also, I am not
      familiar with the mathematical finance literature, so I am not sure how
      interesting this result is within finance context. However, within context
      of online learning this is a novel, and very interesting result.
      1) Please fix the typo ("Marking" -> "Market") in the title of
      the paper -- it's embarrassing :) 2) Please be consistent in
      typesetting authors' names in the references. There are missing dots (e.g.
      "Robert E Schapire"). Sometimes the first name of a person is present,
      sometimes there's just the initial ("S. Das" vs "Sanmay Das").
      Couple of random ideas for future research: - It seems
      that regret of your algorithm depends on the number of switches between
      the underlying base strategies. There are few recent COLT papers with
      variants of Randomized Weighted Majority Algorithm (i.e. Hedge/Exponential
      Weights Algorithm) and Follow the Perturbed Leader that achieve small
      number of switches between experts. (see e.g. "Regret Minimization for
      Online Buffering Problems Using the Weighted Majority Algorithm", COLT
      2010 and "Prediction by random-walk perturbation", COLT 2013). -
      Would it be possibly to get rid of the delta-discretization? The
      discretization seems to be just an ugly artifact of the model/algorithm.
      Or to put it differently, what happens in the limit delta --> 0?


The Pareto Regret Frontier
Performance guarantees for online learning algorithms typically take the form of regret bounds, which express that the cumulative loss overhead compared to the best expert in hindsight is small. In the common case of large but structured expert sets we typically wish to keep the regret especially small compared to simple experts, at the cost of modest additional overhead compared to more complex others. We study which such regret trade-offs can be achieved, and how. We analyse regret w.r.t. each individual expert as a multi-objective criterion in the simple but fundamental case of absolute loss. We characterise the achievable and Pareto optimal trade-offs, and the corresponding optimal strategies for each sample size both exactly for each finite horizon and asymptotically.

This paper concerns regret against individual actions
      in the full-information online prediction setting. Instead of trying to
      bound the regret uniformly for all experts (actions), the work gives
      characterization of what "regret profiles" are achievable for the most
      basic case of binary prediction with absolute loss. The author(s) call a
      profile < r0,r1 > T-realizable if there exists an algorithm that
      guarantees regret r0 and r1 against constant predictions 0 and 1,
      respectively, given any (non-oblivious) adversary. An example for a
      trivial T-realizable profile is < 0,T >. The paper exactly
      (even constants) characterizes the Pareto-front of all T-realizable
      profiles for any T in the basic case of two actions and absolute loss.
      Then they use the result to prove an asymptotic bound. The paper also
      provides algorithms that achieve the realizable profiles. The
      paper is well written, the problem is well motivated, the proofs seem
      sound. The problem of satisfying non-uniform regret bounds is an
      interesting new line of research. This paper takes the first steps in this
      new direction. Some comments: In Definition 1, the
      paper reveals that the adversary model considered is the non-oblivious
      (adaptive) one. I would prefer if it was mentioned more explicitly and at
      an earlier point in the paper. Especially because Lemma 3 does not go
      through for oblivious adversaries. I found the second part of
      Theorem 6 confusing. First I thought it was a full description of the
      algorithm that achieves the vertices of the Pareto-front. Only later I
      realized that it is just some fact about what the algorithm does at three
      particular time steps. Maybe say this explicitly or just leave out this
      part. How do the new bounds relate to the old uniform regret bound
      of \sqrt{T/2 \ln K}? I would have liked to see some discussion about this,
      especially since the above bound is asymptotically optimal. The
      algorithms that prove the upper bounds seem to be highly inefficient
      computationally. It would be nice to have algorithms that are more
      efficient and have regret bounds close to the optimal
      ones.


Distributed Exploration in Multi-Armed Bandits
We study exploration in Multi-Armed Bandits in a setting where k players collaborate in order to identify an ε-optimal arm. Our motivation comes from recent employment of bandit algorithms in computationally intensive, large-scale applications. Our results demonstrate a non-trivial tradeoff between the number of arm pulls required by each of the players, and the amount of communication between them. In particular, our main result shows that by allowing the k players to communicate only once, they are able to learn √ k times faster than a single player. That is, distributing learning to k players gives rise to a factor √ k parallel speedup. We complement this result with a lower bound showing this is in general the best possible. On the other extreme, we present an algorithm that achieves the ideal factor k speed-up in learning performance, with communication only logarithmic in 1/ε.

The paper considers the possibility of parallelizing
      the best arm identification problem. In specific, they ask how and what
      speed-up can by achieved by running k instances of an algorithm in
      parallel if they are allowed to share some information with each other.
      The information sharing is done by each of the instances broadcasting some
      information for the rest of the algorithms (typically at the end of the
      run), but with the restriction that the number of broadcasted bits is
      limited (typically of size \tilde{O}(n) for each instance). The
      authors compare their algorithms to the full-communication method, the
      no-communication method, and the method that takes the majority vote (MV)
      of the individuals recommendations. The first one generates too much
      network data, the second one obviously achieves no speed-up, and, as they
      claim, the MV method achieves no speed-up either. In contrast, the
      algorithm they propose achieves a speed-up of \sqrt{k}. They also show
      that this is optimal if only one round of communication is allowed.
      Finally, they show that if more communication rounds are allowed,
      then there is a trade-off between the possible speed-up (up to k) and the
      amount of information exchanged. EVALUATIION The
      writing style is clear (mostly---but more about that below), the proposed
      solutions are simple but clever, and the problem is both non-trivial and
      reasonable (their motivating example is the distributed computation model
      of MapReduce). However, there are two issues which should be
      clarified. First of all, please define precisely what you mean by
      ``speed-up'', and state your goals formally! It is quite natural to think
      that if some algorithm A achieves confidence 1-\delta in time T, whereas a
      distributed version achieves confidence 1-\delta' during the same time for
      some \delta' much less than \delta, then this is a kind of speed-up.
      (Indeed, for a stand-alone version of A to achieve the same 1-\delta'
      confidence could take much longer than T.) And, in this sense, MV would
      clearly achieve a speed-up: if time T is enough for A to output the best
      arm with probability 2/3, then increasing k to infinity, and simply taking
      the majority vote of the k instances of A (each run for time T) would
      return the best arm with a probability getting arbitrarily close to 1.
      Also, the performance of the MV method should be treated with more
      care. In order to achieve that (in expectation) at least half of the
      players return the best arm, the individual algorithms indeed have to
      output the best arm with probability at least 1/2. However, for the MV
      method to work this is not at all required. In fact, it is enough to have
      that the individual algorithms output the best arm with higher probability
      as any other suboptimal arm. Finally, two less significant
      questions: 1. In the second line of the proof of Lemma 3.3 why do you
      have the multiplicator 6 in front of H/\sqrt{k}? 2. In the second line
      of the proof of Lemma 3.4 shouldn't 12n be devided by \Delta^\star in the
      logarithm?


Learning Stochastic Feedforward Neural Networks
Multilayer perceptrons (MLPs) or neural networks are popular models used for nonlinear regression and classification tasks. As regressors, MLPs model the conditional distribution of the predictor variables Y given the input variables X. However, this predictive distribution is assumed to be unimodal (e.g. Gaussian). For tasks involving structured prediction, the conditional distribution should be multi-modal, resulting in one-to-many mappings. By using stochastic hidden variables rather than deterministic ones, Sigmoid Belief Nets (SBNs) can induce a rich multimodal distribution in the output space. However, previously proposed learning algorithms for SBNs are not efficient and unsuitable for modeling real-valued data. In this paper, we propose a stochastic feedforward network with hidden layers composed of both deterministic and stochastic variables. A new Generalized EM training procedure using importance sampling allows us to efficiently learn complicated conditional distributions. Our model achieves superior performance on synthetic and facial expressions datasets compared to conditional Restricted Boltzmann Machines and Mixture Density Networks. In addition, the latent features of our model improves classification and can learn to generate colorful textures of objects.

This paper proposes a neural network architecture that
      falls somewhere between multilayer perceptrons (MLPs) and sigmoid belief
      networks (SBNs). The motivation is to permit multimodal predictive
      distributions (like SBNs) by using stochastic hidden units, but adds
      deterministic hidden units to smooth the predictive distribution in the
      case of real-valued data. The paper's main technical contribution is an
      EM-style algorithm where the E-step uses importance sampling to
      approximate the posterior and the M-step uses backpropagation to update
      the parameters. The experiments demonstrate the model's utility on several
      synthetic and real datasets. Quality: I liked this paper; the use
      of stochastic and deterministic units seems reasonably justified. The
      model and inference/learning were well presented and I thought the
      experiments were thorough. One criticism would be that the datasets
      considered are either small or subsets of other better-known datasets. In
      other words, large ML baseline datasets are not considered. However, the
      paper makes an effort to compare the proposed model against a number of
      reasonable baselines.Clarity: The paper was generally well
      organized and well-written. It has a number of small grammatical errors
      and would benefit from a thorough proofread. Originality: I am not
      aware of other feedforward neural network architectures that use a mix of
      deterministic and stochastic units. The inference and learning procedure,
      while approximate, seem reasonable (and bear a slight resemblance to the
      sampling methods used in RBM-type models). Significance: The paper
      is correct in that there is often a need to model multimodal predictive
      distributions in machine learning. The proposed model is relatively easy
      to understand and should be able to be easily implemented and extended. I
      think it would be considered by other practitioners.
      CommentsFigure 1 presents a model where the first and last
      layer are wholly deterministic and the inner layers contain some
      stochastic units. Once the reader gets to the experiments, one sees that
      is the type of architecture used in the experiments. But the paper does
      not elaborate on this choice. Is there some significance in choosing a
      first and last fully-deterministic layer, or is this arbitrary?Is
      it necessary to use sigmoid units for both the deterministic and
      stochastic units? Other types of nonlinearities are not mentioned in the
      paper. Page 2, "We note that we can use any other parameterized
      distribution ... for the output variables. This is a win compared to the
      Boltzmann Machine family of models, which require the output distribution
      to be from the exponential family." Is this totally true? I agree that
      exponential family Boltzmann machine models are convenient and the norm,
      but isn't one completely free to define the energy function and interpret
      the distributions however one chooses?The baseline SBN considered
      in the experiments is trained in the exact same way as the SFNN, only it
      has no deterministic units. Have you compared to Sigmoid Belief networks
      trained in other ways? Is there an advantage to training them with the
      proposed algorithm designed for SFNNs? That would be an interesting
      secondary result.The experiments in section 3.1 do not say how the
      test log-probability was approximated for the C-GRBM (which has an
      intractable partition function, so log likelihood cannot be computed
      exactly). The following section says that AIS was used; is this also the
      case for section 3.1?Figure 5 shows how different hyperparamters
      affect the evaluation and learning of SFNNs, including the absolute number
      of stochastic nodes per layer. I'd be interested in seeing the ratio of
      stochastic to deterministic nodes, rather than the absolute number of
      stochastic nodes.Minor comments- Footnote of pg 1, \sigma
      is overloaded (used for both variance e.g. \sima_y^2 and also to mean the
      sigmoid function)- Inline math immediately after Eq 1, bias_2, bias_1
      is ugly notation; why not use a lowercase letter for the
      biases?


Approximate Inference in Continuous Determinantal Point Processes
Determinantal point processes (DPPs) are random point processes well-suited for modeling repulsion. In machine learning, the focus of DPP-based models has been on diverse subset selection from a discrete and finite base set. This discrete setting admits an efficient sampling algorithm based on the eigendecomposition of the defining kernel matrix. Recently, there has been growing interest in using DPPs defined on continuous spaces. While the discrete-DPP sampler extends formally to the continuous case, computationally, the steps required are not tractable in general. In this paper, we present two efficient DPP sampling schemes that apply to a wide range of kernel functions: one based on low rank approximations via Nyström and random Fourier feature techniques and another based on Gibbs sampling. We demonstrate the utility of continuous DPPs in repulsive mixture modeling and synthesizing human poses spanning activity spaces.

This paper extends determinantal point process (DPP)
      sampling schemes from discrete to continuous spaces by exploiting
      low-rank kernels. Standard kernels are approximated using the Nystroem
      or random Fourier features methods. The method is demonstrated on
      repulsive mixture models and separated sampling. The paper
      exploits a low rank kernel in order to derive an efficient "dual
      sampler", as described in Alg 1. The authors show in which cases of
      quality function and similarity kernel the necessary computations can
      be carried out for the RFF and Nystroem approximations (Supp tab 1).
      This is a non-trivial piece of work. In sec 5 the continuous DPP
      is used as a prior on the means in a mixture model. However, note that
      the approximation derived before is not actually used; as the authors
      say "since we are sampling just one item from the DPP there is a
      closed form and we do not rely on approximations." The results show
      that DPPs do (as expected) have a repulsive effect, as evinced by
      the mixture weight entropies. Aside: surely to qualify as a
      repulsive mixture, the closeness of the means should really be
      relative to their variances, but in supp mat p 4 the variances are
      independent of the means ... In sec 6 a DPP is used to sample a
      diverse set of poses from a given density. While the resulting poses
      are credibly diverse, I suspect that visually similar results could
      have been obtained e.g. using hard core processes (see p 1), or some
      scheme like farthest point clustering (Gonzales, 1985).
      Quality: As far as I can judge the material is technically
      correct. Clarity: Generally clear. As to Fig 1, I wanted
      more detail as to the defn of (15) -- for a given set X we can
      evaluate this, but how is the sum defined and evaluated?
      Originality: as the authors make clear, this work extends earlier
      work on low-rank approximations in discrete spaces. And as they
      mention the repulsive prior construction has been used before [20,32].
      Significance: it is not very clear about the applicability of the
      main result. Sec 5 does not actually use it, and sec 6 is just a
      visualization which does not require a precise sampler. Of course it
      is possible that the result could be exploited in other interesting
      ways. Other: * l 102. [11] are not the first to
      show that a matrix with the form B^T B (D << N) has a fast
      eigendecomposition, while the form of the citation implies it is. For
      example this result was used in L. Sirovich and M. Kirby (1987).
      "Low-dimensional procedure for the characterization of human faces".
      Journal of the Optical Society of America A 4 (3): 519–524, but is
      surely in any good advanced linear algebra text, e.g. Wilkinson?
      * in eq 2 define the \bar{ \phi_n} notation (presumably as complex
      conjugate). * l 241-242 why not sigma^2 I instead of
      diag(sigma^2, \ldots, sigma^2), and the same with rho^2? * l
      304 -- why use only 100 test observations? Reduce variance by using
      many more!


Streaming Variational Bayes
We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data-a case where SVI may be applied-and in the streaming setting, where SVI does not apply.

The authors present a Streaming Distributed
      Asynchronous (SDA) framework for variational Bayesian computation aimed at
      analysis of large datasets. They assume a variational form q(θ) for the
      posterior on model parameters that is exponential family, which they
      express as q(θ) ∝ exp[ξ . T(θ)]. Note that this parameter vector ξ = (χ ν)
      contains both the cumulant of the conjugate sufficient statistics χ and
      the scale parameter ν. The authors then define a set of parameters ξ_0 for
      the variational approximation of the prior p(θ), and make the observation
      that when the data is split up into batches, independent posteriors ξ_b
      for each batch may be combined to obtain a approximation for the posterior
      on the aggregate data ξ = ξ_0 + Σ_b (ξ_b - ξ_0). Τhey propose an algorithm
      where each new batch is used to update ξ_0, i.e. for each new batch prior
      is taken to be the posterior after observation of the previous batch. Τhis
      algorithm can be run in a distributed setting where each worker returns
      Δξ_b = (ξ_b - ξ_0) to the master, which then updates its copy of ξ_0 <-
      ξ_0 + Δξ_b accordingly. Τhe proposed methodology is simple but
      useful, primarily because it presents a trivial parallelization strategy
      that may be applied to any algorithm that obtains an exponential family
      estimate of a posterior p(θ | x). The main competitor to this method is
      stochastic variational inference (SVI), a method that uses a stochastic
      optimization based on a gradient estimate of the lower bound evidence. A
      disadvantage of SVI, as the authors note, is that it is sensitive to
      parameters for the total number of expected documents, batch size, and
      step size. However I am not entirely convinced this is always a bad thing.
      While tuning parameters can indeed be costly, having such tunable
      parameters also makes a method more flexible. The presented results on LDA
      are promising, but they are not sufficient to demonstrate that SDA can
      match or exceed SVI results in a variety of domains while eliminating the
      need for such tunable settings. I would have also liked to have
      seen a clearer motivation for the proposed distributed update scheme. When
      comparing their chosen update scheme with one where ξ_0 is held constant
      for each of the workers, the authors note that the "key difference between
      the first and second frameworks proposed above is that, in the second, the
      latest posterior is used as a prior. This latter framework is more in line
      with the streaming update of Eq. (2) but introduces a new layer of
      approximation". Unfortunately the implications of this approximation are
      left unexplored, save for the otherwise unqualified statement that "we
      find that the latter framework performs better in practice, so we focus on
      it exclusively in what follows". It would seem that the essential
      difference between SDA and SVI is that in SDA ξ grows with the number of
      documents, whereas the variational parameters have a fixed strength in
      SVI. This implies that in SDA (1) the entropy of the posterior q(θ)
      decreases as more data is seen, and that (2) convergence is ensured by
      virtue of the fact that in later updates (ξ_b - ξ_0) << ξ_0. In SVI
      convergence is achieved by annealing the step size to 0 and ξ has a fixed
      strength determined by the total number of documents D, which in an
      on-line setting one might interpret as a buffer size. It would be
      interesting to see a more detailed analysis of which of these two
      strategies is more effective in terms of ensuring rapid convergence while
      avoiding local maxima. In short I believe this may be a promising
      approach to variational inference in large data sets, chiefly because it
      is by design suitable to distributed inference in a map-reduce type
      setting. However I do feel more thorough testing is needed before this
      method can be considered a drop-in replacement for SVI. --
      Minor points / Questions • I see no reason why SVI could not
      be performed with distributed updates precisely in the manner proposed
      here for SDA. Have the authors considered this scenario? • Are the
      authors using their own implementation of SVI in the analysis comparison,
      Matt Hoffman's code
      (http://www.cs.princeton.edu/~mdhoffma/code/onlineldavb.tar) or the Vowpal
      Wabbit implementation (http://hunch.net/~vw)? Given that a run-time
      comparison is performed, it would be desirable to state this explicitly
      • The authors write that in SVI "the stochastic gradient is
      computed for a single data point (document) at a time". This statement is
      technically correct but somewhat confusing, since it suggests SVI is
      performed on single data-points, not batches.


Graphical Models for Inference with Missing Data
We address the problem of recoverability i.e. deciding whether there exists a consistent estimator of a given relation Q, when data are missing not at random. We employ a formal representation called 'Missingness Graphs' to explicitly portray the causal mechanisms responsible for missingness and to encode dependencies between these mechanisms and the variables being measured. Using this representation, we derive conditions that the graph should satisfy to ensure recoverability and devise algorithms to detect the presence of these conditions in the graph.

Review Summary: This paper presents a graphical models
      approach to reasoning about when probabilities can be computed in the
      presence of missing data. The approach hinges on definitions of MCAR and
      MAR in terms of conditional independencies between data and response
      indicator variables. The MAR definition is considerably stronger than the
      standard definition studied by Little and Rubin, which encodes the minimal
      assumption required to obtain unbiased estimates while ignoring the
      missing data mechanism. The current work is certainly novel and
      interesting. It appears to be technically correct given the initial
      definitions. However, the main component the paper is missing is a
      discussion of the impact of changing the definitions. Detailed comments
      are provided below. Novelty/Originality: The work appears to be
      novel. The fact that the authors are able to provide a treatment of the
      missing data taxonomy using graphical models is interesting and stems from
      their MCAR/MAR definitions based on independence between random variables.
      The standard definition of MAR is a statement of symmetry within the
      distribution P(R|X) that depends on the actual values of R and X. It is
      not a statement of independence between random variables, so Bayesian
      networks can not be used to describe it. Technical Correctness:
      Given the initial MCAR/MAR/MNAR definitions, the remaining technical
      development appears to be correct. The authors show that they can apply
      their framework to reason effectively about what queries are recoverable,
      even in some fairly complex situations. The main issue with the
      paper is the definitions themselves. Defining MAR at the random variable
      level means that data sets that would be considered MAR by Little and
      Rubin could correspond to graphs considered MNAR here. This creates some
      confusion with the claims in the paper. The authors repeatedly
      make claims like "Though it is widely believed that relations in MNAR data
      sets are not recoverable, [we] demonstrate otherwise" (lines 229-230).
      This is confusing since the majority of work uses the standard definition
      of MNAR from Little and Rubin, which differ from the definitions used
      here. Are the authors saying that under *their* definition of MNAR, it is
      widely believed that queries are not recoverable? Additionally,
      the authors should clarify what the relationship is between the the
      recoverable MNAR cases under their framework, and how data from the
      corresponding models would be labeled by Little and Rubin's definitions.
      For example, is it the case that all of the MNAR graphs presented here
      that admit recoverable queries would generate data that Little and Rubin
      would consider MAR? Are any of the examples shown graphs that are NAMR
      under the authors' definition, but admit recoverable queries, and would
      also generate data considered MNAR by Little and Rubin?
      Significance/Impact: The significance of the work is difficult to
      judge without the clarifications requested above. There could be some
      interest in these alternative definitions and the example applications of
      the framework are quite nice. On the other hand, the limitations have not
      been described by the authors at all. An additional issue with the
      paper is the question of where the structures that are reasoned about come
      from? Given a novel domain, coming up with a model over the primary
      variables can be challenging. The additional complexity to soliciting a
      joint data/response model would be quite onerous. Given possibly NMAR
      data, what are the prospects for learning these model structures? Given
      two candidate structures, one in which a given query is recoverable and
      another where it is not, can the framework itself provide any guidance on
      which structure to choose? Presentation/Clarity: The paper is well
      written although quite dense. Focusing the development around specific
      examples does help, but the examples themselves become quite complex.
      * The name of the model is introduced twice on page 3 (lines 80,
      115, 135). * In equation 1, what does the value m stand for? It is
      not introduced prior to this point in the paper. I assume it stands for
      "missing"? * I would consider a more politically correct example
      on line 149.