Multilabel Structured Output Learning with Random Spanning Trees of Max-Margin Markov Networks
We show that the usual score function for conditional Markov networks can be written as the expectation over the scores of their spanning trees. We also show that a small random sample of these output trees can attain a significant fraction of the margin obtained by the complete graph and we provide conditions under which we can perform tractable inference. The experimental results confirm that practical learning is scalable to realistic datasets using this approach.

***** Update after author feedback at the end ********************************************************quality:  8 (out of 10)clarity: 7originality: 6significance: 7SUMMARY: The authors consider the use of random spanning trees to do structured prediction when the underlying graph structure on the labels is unknown. They motivate their approach by first showing that under some assumptions, the discriminant function over a fully connected graph on the labels can be expressed as the uniform expectation of the discriminant functions over random spanning trees. Through a sampling result, they then show that with high probability over samples of random spanning trees, there is a conical combination of these trees which achieve a substantial fraction of the margin of a predictor which uses the complete graph, and then prove a related risk bound for conical combination over random trees. This motivates to optimize the margin for conical combination of trees as predictors, and the author proposes a primal (and dual) formulation for this optimization problem (somewhat analog to the structured SVM), for which a standard dual subgradient method is proposed as in previous work. They then show that the maximizing joint label for the combination of trees (inference) can be done exactly (under an assumption that be checked at run-time) by looking through the K-best list for each spanning tree (the latter can be obtained by dynamic programming, as was already mentioned in Tsochantaridis et al. [JMLR 2005]). Experiments on standard multilabel datasets show a small improvement over alternative methods. Moreover, the authors investigate the proportion of times they can get an exact solution (according to the checkable condition they derived) as a function of K and the number of trees on the datasets. On some datasets, the exact solution is obtained a large fraction of time even for small K.EVALUATION:The authors claim in their introduction (line 051) "Our work is not based on any of these approaches, despite superficial resemblances to (for example) the trees in tree-based relaxations and the use of random trees in [5]." But I see this paper as a joint learning / joint inference formulation of [5] with a well-principled theoretical motivation (vs. the combination heuristics proposed in [5]). There is a consistent theoretical story behind the learning framework which is interesting (and I appreciate that all the proofs [except for theorem 5 -- see below] are provided. The paper is fairly clear to read and well-organized;  on the other hand, there are several technical assumptions made for which it unclear how crucial they are for their results. They also seem more geared towards the multilabel setup than a general structured prediction problem. For example, they only consider the 0-1 loss in the theory (and not Hamming e.g.); and they assume that the weight vectors are not shared across the edges in the graph (unlike is often the case in typical structured prediction applications). They also assume that the features are normalized on each edge. How important are these assumptions for their results? I would like the authors to comment on this in their rebuttal.I am not knowledgeable of the multilabel classification applications, so I cannot judge how significant are their empirical results. But the methodology seems sound; and it was interesting to observe that often the K-best lists were sufficient to obtain the global max.Pros:-A fairly complete theory motivating their learning objective.- Combinations of random spanning trees seem like a reasonable approach to avoid working with the full graph when the graph is unknown given their explanations.- Reasonable experimental comparison with somewhat practical algorithms.Cons:- Several technical assumptions are made along the way for which it is unclear how important they are to their results (or they are just there for simplicity).- The authors don't really relate their work to the existing literature apart saying that the resemblances are "superficial". == Other comments ==- Line 204: they cite [8] for the risk bound -- but in [8] (e.g. Theorem 4.17), the risk bounds are only for binary labels as far as I could find. How do the authors adapt it to the exponential number of multilabels? Can they provide a more specific pointer?=== Update after author feedback === Thank you for the clarifications.My comment about the technical assumptions was to clarify which conditions are for simplicity of presentation or simplifying the statements, and which ones are essential to the current proofs. Indeed, I was referring to the normalization assumption for the feature map. In particular, a different normalization could be used in practice (even though "it can be performed for most phi"); and so could a different normalization be easily handled? Similarly, weight sharing and non 0-1 loss are important in structured prediction applications; how hard would it be to extend the results to these settings? The authors should clarify this in their final version (as well as the other aspects that they said they would clarify).


Divide-and-Conquer Learning by Anchoring a Conical Hull
We reduce a broad class of fundamental machine learning problems, usually addressed by EM or sampling, to the problem of finding the k extreme rays spanning the conical hull of a1 data point set. These k "anchors" lead to a global solution and a more interpretable model that can even outperform EM and sampling on generalization error. To find the k anchors, we propose a novel divide-andconquer learning scheme "DCA" that distributes the problem to O(k log k) sametype sub-problems on different low-D random hyperplanes, each can be solved independently by any existing solver. For the 2D sub-problem, we instead present a non-iterative solver that only needs to compute an array of cosine values and its max/min entries. DCA also provides a faster subroutine inside other algorithms to check whether a point is covered in a conical hull, and thus improves these algorithms by providing significant speedups. We apply our method to GMM, HMM, LDA, NMF and subspace clustering, then show its competitive performance and scalability over other methods on large datasets.

Summary: This work proposes a new algorithm, "divide-and-conqueranchoring" (DCA), that solves the problem of selecting a subset A ofrows from a matrix Y such that those rows best model another matrix X:X = FY_A.  Previous work [29] addresses the simpler setting where X =Y and shows how this setting generalizes non-negative matrixfactorization (NMF).  The even more general X != Y setting that thiswork tackles is shown to generalize many core machine learningproblems: Gaussian mixture models (GMMs), hidden Markov models (HMMs),latent Dirichlet allocation (LDA), and subspace clustering (SC).  Thissetting relies on a "general separability assumption", which requiresthat there exist a subset A of Y's rows such that the rows of X arecontained in their cone.  Lemma 1 gives a very simple condition forthe identifiability of A, which allows DCA to be much more robust tovariance in the sample moments than spectral methods.  Experimentalresults on a large number of tasks indicate that DCA is almost alwaysmuch faster than traditional learning methods (including spectralones) without losing significant accuracy.Comments:Clarity, quality: The general idea and algorithm of the paper isclear, but the math in this work is pretty dense.  Additional small,illustrative examples and figures (similar to that of Figure 2) wouldbe helpful to make some of the concepts more concrete.Originality: The work of [29] seems to be the most closely related tothis paper.  The X != Y difference represents a large delta in termsof theoretical and experimental work though, so this paper iscertainly sufficiently original.Significance: Given the large number of practical machine learningtasks that this minimum conical hull problem encompasses, and itsclear performance advantages on them, this work seems verysignificant.Supplement: It's somewhat difficult to map back and forth between thedocument and the supplement, since the theorem/lemma/etc. numbersdon't match.  When referring to [3] (the supplement) from the mainpaper, it would be helpful if the authors could use the optionalargument of the cite command to indicate more precisely which part of[3] is being referenced: \cite[specific location]{supplement}.Additionally, there is a lot of information in the supplement thatisn't represented in the main body of the paper.  Perhaps this paperwould be better suited to a venue without page limits (such as ajournal).Experiments: Do the authors have plans to release the associated code?How were the s-values (50, 92, 133, 175) selected?  Often DCA, winsspeed-wise but loses slightly accuracy-wise.  Can the authors commentfurther on why this might be the case?  Are there knobs that can butturned on the DCA algorithm to alter the speed/accuracy balance?(Besides the s-knob, which doesn't seem to have a huge effect.)Organization: Since NMF is already covered by [29], it might makesense to leave its explanation and experiments out of the main body ofthe paper and focus on explaining the other tasks (or the theory) ingreater detail.The variable "c": The variable "c" in the definition of "s" on line268 is only really defined in the supplement.  Even there it'ssomewhat unclear what c's range is.  Can the authors provide amathematical statement describing the relationship between c and theflatness of the flattest anchor?  Is n the maximum value for c?Projection matrices: Proposition 1 indicates that projection matricesare drawn uniformly from the Grassmanian manifold, and the supplementindicates several other random matrix ensembles that could be drawnfrom.  It would be helpful if the authors could provide a fewreferences detailing how to draw from each of these.More minor notes:1) "although spectral method using" -> "although spectral methodsusing"2) "random drawn low-dimensional" -> "randomly drawn low-dimensional"3) Equation (1) and elsewhere: Use \left and \right to make bracketsand parentheses match the height of the math that they enclose.2) "general models such as matrix factorization and latent variablemodel" -> "general models such as matrix factorization and latentvariable models"4) "most popular models such as GMM and HMM" -> "most popular modelssuch as GMMs and HMMs"5) "the two D matrices in RHS of" -> "the two D matrices in the RHS of"6) "either matrix in LHS of" -> "either matrix in the LHS of"7) "each handled by a solver to minimum conical hull problem" -> "eachhandled by a solver to the minimum conical hull problem"8) "anchoring algorithm for sub-problem on 2D-plane" -> "anchoringalgorithm for a sub-problem on the 2D-plane"9) "Due to the convexity of cone" -> "Due to the convexity of cones"10) "For arbitrary point" -> "For an arbitrary point"11) "w.h.p.  solving" -> "w.h.p.\ solving"12) "given two sets of points(rows)" -> "given two sets of points(rows)"13) "under general separability assumption" -> "under the generalseparability assumption"14) Supplement, line 415: "Lemma ??" -> "Lemma 2"15) "each point on each curve is the result by averaging" -> "eachpoint on each curve is the result of averaging"16) "Because DCA uses separability assumption" -> "DCA uses theseparability assumption"17) There are many grammatical errors in the supplement besides theones listed above.  It could use a more careful proofreading.


Rates of convergence for nearest neighbor classification
We analyze the behavior of nearest neighbor classification in metric spaces and provide finite-sample, distribution-dependent rates of convergence under minimal assumptions. These are more general than existing bounds, and enable us, as a by-product, to establish the universal consistency of nearest neighbor in a broader range of data spaces than was previously known. We illustrate our upper and lower bounds by introducing a new smoothness class customized for nearest neighbor classification. We find, for instance, that under the Tsybakov margin condition the convergence rate of nearest neighbor matches recently established lower bounds for nonparametric classification.

This paper develops a new way of analyzing k-Nearest Neighbor prediction for classification problems. Nearest Neighbor prediction is a simple and well studied classification method that, although the first results date decades back, has recently seen some renewed interest and new development. This paper provides a new angle on Nearest Neighbor analysis by incorporating that the behavior of NN classifiers actually automatically adapt to local scaling (with respect to the probability distribution) of the input space. This aspect has not been taken into account in previous studies on NN.Previous analysis of NN has been in terms of smoothness parameters and bounds provided reflect some sort of worst case over the data space of this smoothness parameters. Bound derived like this can be rather loose. This study aims to overcome this drawback. The authors define a new notion of margin (that depends on k and n) and bound the expected error (with repect to the Bayes) in terms of the probability weight of this margin. This is complemented with a lower bound in terms of a similar notion. As a result, the authors obtain a general consistency result as well as some bounds under additional assumptions.The drawback of the paper is that most of the results have a bit of a “in the middle of the analysis” feel (with the exception of Section 2.6). It is not clear how useful it is to say that the error is bounded by the weight of some subset of the domain, when there is no handle/estimate on how much weight there is on that set...However, I do think that this work provides sufficient novel insights on the behavior of NN classifiers to justify publication at NIPS. Moreover, the paper is a nice read.


Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
We introduce a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. We then introduce a structured max-margin objective that allows our model to explicitly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions for the image-sentence retrieval task since the inferred inter-modal alignment of fragments is explicit.

This paper proposes a novel approach to learning joint image-text embeddings for the tasks of image to text retrieval and vice versa. The authors propose the use of image and sentence fragments, which come from convolutional net features of detected objects and dependency tree relations. A ranking objective is proposed that combines both a fragment and global objective, where latent dependency-fragment alignments are inferred with a MIL extension. Extensive experimentation is performed on the ranking tasks proposed by Hodosh et al, including results on the new Flickr30k dataset.This is a nice paper. The authors obtain strong performance on each of the datasets and the experimentation is well done. Model choices are experimentally verified with convincing results. I have some minor questions / comments below that I hope the authors can respond to:- lines 018-021: While this work is the first to go beyond a global ranking objective, conceivably even previous methods that work globally should be able to reason locally. For example, each intermediate node of an RNN would live within the same embedding space (as well as words, if they are fine-tuned). This would allow, for example, the retrieval of images from a single word or short phrase, even through training was only done globally. Of course, I would expect better results from your approach since such fragment-level embeddings are being optimized for directly.- line 137: Wouldn't W_e be d x 400,000, since each of w_1, w_2 is 400,000 x 1?- It would be helpful to summarize the space of hyperparameters that you tune (there seems to be a lot of them). Do the best validation hyperparameter choices vary greatly across datasets? Any intuition about the hyperparameter space (are some choices consistently better than others, etc) would be helpful for the reader to gain some intuition. How sensitive are the validation results to \beta (the weight on the global objective)?- On a related note, even though you were not able to cross-validate all the hyperparameters of Flickr30k with SDT-RNN, perhaps you could report results on Flickr30k using the best selected hyperparameters from Flickr8k (unless there is overlap between the two datasets)- section 4.1 baselines: Are you also including the CNN features for the full image, or just the detected objects? - You mention that the class predictions from the convolutional net are discarded. Wouldn't these have been useful to inferring the fragment alignments?- One of the challenges of this task is obtaining large enough datasets with detailed, accurate image captions. The experimental results are convincing in this case, since the three datasets used went through an extensive annotation process. How do you think your method would do on larger, but less accurate captioned datasets? For example, the SBU captioned photo dataset has 1 million images but many of these captions do not necessarily depect visual scenes (e.g. 'fido' referring to a dog). On such datasets, I would suspect that the use of fragment embeddings would be less beneficial.


Optimal Teaching for Limited-Capacity Human Learners
Basic decisions, such as judging a person as a friend or foe, involve categorizing novel stimuli. Recent work finds that people's category judgments are guided by a small set of examples that are retrieved from memory at decision time. This limited and stochastic retrieval places limits on human performance for probabilistic classification decisions. In light of this capacity limitation, recent work finds that idealizing training items, such that the saliency of ambiguous cases is reduced, improves human performance on novel test items. One shortcoming of previous work in idealization is that category distributions were idealized in an ad hoc or heuristic fashion. In this contribution, we take a first principles approach to constructing idealized training sets. We apply a machine teaching procedure to a cognitive model that is either limited capacity (as humans are) or unlimited capacity (as most machine learning systems are). As predicted, we find that the machine teacher recommends idealized training sets. We also find that human learners perform best when training recommendations from the machine teacher are based on a limited-capacity model. As predicted, to the extent that the learning model used by the machine teacher conforms to the true nature of human learners, the recommendations of the machine teacher prove effective. Our results provide a normative basis (given capacity constraints) for idealization procedures and offer a novel selection procedure for models of human learning.

The work presented here is solid and innovative from a cognitive scienceperspective, and the manuscript is well written and for the most part clear.  Ihave some issues and questions for the authors, which I suspect can beaddressed.Major comments:On line 031 of the abstract, the authos conclude, "We find that the optimalteacher recommends idealized training sets."  I believe that the idealizationof training sets is a presumption of the 'optimal' teacher, so the teacher isreally an 'optimal idealized-training-set' teacher.  I would be happy if theauthors simply stated at the outset that they were going to limit theirexploration to a search for idealized training sets, based on pastpsychological research.  Because they seem unwilling to make such a statement,they introduce idealization in a slippery backdoor kind of way (see line 175,the sentence beginning "We do not have evidence...")It is true that the various idealized training sets outperform one particularnonidealized set (the random set), but I don't think the right comparisonshave been done to conclude that idealization is the optimal strategy forlimited-capacity human learners.It would be useful to say a few words about what relevance idealization has totraining real-world tasks.  Idealization is a different idea than simply usingprototypes: idealization involves labeling nondeterministic stimuli with theirpredominant classification.  But with naturalistic tasks, one may not be ableto ascertain the degree of ambiguity of a stimulus.  Although the paper is notabout idealization per se, it presumes that the teacher can idealize stimuli.The authors report that the optimal teacher selects clump-far examplesfor a limited-capacity learner (line 222). I would like to know more aboutthese clump-far sets.  Figure 2A shows that the 'far' clump is part way but notall the way to the stimulus dimension extrema.  Why not stimuli furtherfrom the boundary?  What's the trade off that lands the 'far' stimuli at around.15 and .85?  Is the optimal set really 5 identical examples adjacent to5 identical examples, or is that simply how the figure is drawn?I didn't go back and look at the Nosofsky & Palmieri Psych Rev paper (reference[17]), but it's difficult to intuit the claim that \gamma relates to memorycapacity (line 110).  My intuitions are further shot by the comment (lines227-299): "The high-capacity GCM...is sensitive only to the placement oftraining items adjacent to the decision boundary".  This doesn't sound like ahigh capacity model.Until the last paragraph of the article (line 429), the authors did nottouch on an issue that bothered me from the outset: that ordereffects are ignored.  To the degree that human memory is capacity limited,it is also recency biased, and the fact that recency doesn't come into playfurther weakens my acceptance of the low-lambda GCM as a model oflimited-capacity instance-based learning.MINOR COMMENTS:line 025: In the abstract and the introduction, the authors comment thatidealization runs counter to ML practice, where one aims to match statistics ofthe training and test sets.  However, with limited training set size, my hunchis that idealization would benefit ML classifiers as well on the particulartask studied in this paper.  For example, comparing a 'spread' idealizedtraining set to a 'spread'  non-idealized set, It seems likely that thevariability in the inferred decision boundary will be smaller with theidealized set.057: What does "idealized in an ad hoc or heuristic fashion" mean?192, "discuss the experiment with human learners": There's ambiguity in theprepositional phrase attachment. You might want to re-word.205: I'm lost on the notation. Where are the empirical probabilities shown?Is the $y_j = 1$ term in the equation for $f^{(cond)}$ the observed outcomeof an experimental trial?Figure 4: It might be a good idea to zoom in to focus on the 50-75%performance range.


Predicting Useful Neighborhoods for Lazy Local Learning
Lazy local learning methods train a classifier "on the fly" at test time, using only a subset of the training instances that are most relevant to the novel test example. The goal is to tailor the classifier to the properties of the data surrounding the test example. Existing methods assume that the instances most useful for building the local model are strictly those closest to the test example. However, this fails to account for the fact that the success of the resulting classifier depends on the full distribution of selected training instances. Rather than simply gathering the test example's nearest neighbors, we propose to predict the subset of training data that is jointly relevant to training its local model. We develop an approach to discover patterns between queries and their "good" neighborhoods using large-scale multilabel classification with compressed sensing. Given a novel test point, we estimate both the composition and size of the training subset likely to yield an accurate local model. We demonstrate the approach on image classification tasks on SUN and aPascal and show its advantages over traditional global and local approaches.

This paper proposes a method to predict, given a novel test instance, a useful subset of examples to train a local classifier. Existing methods in the space of lazy local learning typically fix the size of the selected subset and select relevant training instances based on the proximity (nearest neighbors) to the test example. By contrast, the proposed method estimates the required size of the relevant subset while allowing for deviation from a strict nearest neighbor ranking. The method consists of three main steps. The first (offline) is to generate a training set of instance-neighborhood pairs (obtained by distance-weighted sampling of neighbordhoods of varying sizes and for each training intance, heurestically selecting the best performing such pairs). The second (also offline) is to rely on the multi-label classification framework of [19] to learn a mapping between an instance and its useful neighborhood (reperesented as an indicator vector of confidences over the entire training set). The third (online) applies the learned mapping to a novel test intance to predict an indicator vector of confidences: from this vector, candidate neighbordhoods, of varying sizes, are generated via confidence-weighted sampling and used to train a number of local classifiers. Finally the prediction of the most confident local classifier is retained.This paper is extremely well written. The ideas are presented very clearly. The authors constantly situate the paper and the proposed methods well within the existing body of works. The result is a paper that reads well and highligths the important differences with respect to existing works.My main concern at this juncture is the fairness of the experimental evaluation. In general, the reviewer finds it surprising that the local baselines perform so poorly (almost consistently at chance) on the SUN dataset given that it is indeed perfectly well suited for lazy learning. However, beyond this, in the current setup it is unclear wether the gains with respect to the local baselines are due to the learning of relevant neighborhoods (steps 1 and step 2) or whether they are simply due to choosing the most confident local classifier out of a presumably large (how large is it?) set of trained classifiers (the final step in step 3). In other words, the reviewer believes a key and important baseline is missing here. This baseline would consist of applying step 3 on a confidence indicator vector (y^_q) whose entries are proportional to the distance (eucledian or learned) to the test query. In this manner, we can evaluate whether the reported gains are due to the learned y^_q or whether they are due to the trial and error tests of step 3, in which case the contribution of the paper is very small.In addition, the reviewer does not believe that using the auto-selected k from the proposed method to evaluate the local baseline is fair. The current reported local baselines should have their own auto-selected k values. If this is the case, please  clarify.Also, again with regards to experiments, it would be very important (even critical) to make explicit the exact number of positive training instances per data set and per attribute as well as the distribution of k per attribute so that we may get an understanding of the proportion of positives that are selected.Finally, the reviewer would like a motivation for the computation of k* in line 175 via an averaging of probabilities (in general probabilities cannot be averaged) for step 1. Also, how is the Platt calibration performed exactly? Finally, are the local classifiers in step 3 calibrated? If so, how? How many such local classifiers are used?


Magnitude-sensitive preference formation
Our understanding of the neural computations that underlie the ability of animals to choose among options has advanced through a synthesis of computational modeling, brain imaging and behavioral choice experiments. Yet, there remains a gulf between theories of preference learning and accounts of the real, economic choices that humans face in daily life, choices that are usually between some amount of money and an item. In this paper, we develop a theory of magnitudesensitive preference learning that permits an agent to rationally infer its preferences for items compared with money options of different magnitudes. We show how this theory yields classical and anomalous supply-demand curves and predicts choices for a large panel of risky lotteries. Accurate replications of such phenomena without recourse to utility functions suggest that the theory proposed is both psychologically realistic and econometrically viable.

This paper presents a rational model of economic choice, where values are replaced by inferences about relative rank in a context. The model is shown to provide a good fit to pricing data and risky-choice experiments.Overall, I found the paper interesting, but somewhat too narrow and inaccessible for a NIPS audience. There was lots of insufficiently explained economic jargon (e.g. Veblen and Giffen goods), which are key for understanding the contribution. Moreover, the point of the model was hard for me to pull out of the paper. The model was motivated as psychological realistic alternative to utility calculations, but then uses Bayesian inference as a psychological calculation (not as rational model). Other points:-Good intro, but a bit too aggressive against the values of utilities. Well-written and fairly clear.-Variables not obvious in equations. For example, o seems to be observations, but that is never mentioned.-To reiterate, something not above, I was very confused about the point of this model. It was motivated as psychological realistic alternative to utility calculations, but then uses Bayesian inference as psychological calculation (not as rational model). I don’t see how this improves over the Decision-by-Sampling model, which provides a more realistic psychological grounding and already covers the some of the key features oWhat is the contribution? Is it the rational model/analysis? -Hanging a lot hang on number frequency means there should be all sorts of weird quirks at large, common numbers (e.g., $100 vs $101). Is there any evidence or data that these frequency anomalies exist? And, I don’t see how this guarantees a Pareto distribution? Little support is given to the key claim that number space will follow this distribution, and, intuitively, I don’t see it all. The numbers are certainly not strictly descending in frequency in many contexts. Imagine buying car. $10 is a lot less frequent than $15000. Finally, there is a lot of work on subjective number representation that is not cited here, but also very relevant. Number (as do most senses) follows Weber’s law, whereby larger numbers are less discriminable than smaller numbers (think 1 vs. 2 or 200 vs. 201; see, for example, Dehaene, 2003, in Trends in Cognitive Science). - Figures 2 and 3 had tones of information, but very little guidance, and I had lots of trouble making sense of them. For example, in Figure 2, what is m (in s, m, d) in top two panels on left? Normal? Not n? And what are the three plots on the topmost panel (presumably hill, valley, and forest, but that’s not labeled). Also, in Fig 2., I like the attempt to explain the pieces of the equation intuitively, but was admittedly baffled by what the bottom 3 panels on the right are supposed to represent. pmf = ? The text only covers some of the figure and is thus unhelpful. Minor points:line 080: direct utile estimation = ?line 082: “from the history of past choices” (missing article)line 107: identitied?lines 145: sampling a representative context sounds far from simpleline 151: could be more simply stated in words than the equation, to wit: “when x is the most preferred option in a context”. At the very least, unpack the equation.line 194. “in via”line 267. [?]line 363. Is q or p(x) the lottery probability?line 377. computatationline 409-411. A bizarre claim to make about RL and back up with the Sutton & Barto book.Ref 19 is mangled: The Sutton & Barto book (not Barto & Sutton) was published by MIT press, which is in Cambridge, Mass. (not U of Cambridge).


Efficient learning by implicit exploration in bandit problems with side observations
We consider online learning problems under a a partial observability model capturing situations where the information conveyed to the learner is between full information and bandit feedback. In the simplest variant, we assume that in addition to its own loss, the learner also gets to observe losses of some other actions. The revealed losses depend on the learner's action and a directed observation system chosen by the environment. For this setting, we propose the first algorithm that enjoys near-optimal regret guarantees without having to know the observation system before selecting its actions. Along similar lines, we also define a new partial information setting that models online combinatorial optimization problems where the feedback received by the learner is between semi-bandit and full feedback. As the predictions of our first algorithm cannot be always computed efficiently in this setting, we propose another algorithm with similar properties and with the benefit of always being computationally efficient, at the price of a slightly more complicated tuning mechanism. Both algorithms rely on a novel exploration strategy called implicit exploration, which is shown to be more efficient both computationally and information-theoretically than previously studied exploration strategies for the problem.

Summary:This paper considers the multi-armed bandit problem with side information, previously studied by Alon et al. [1] and Mannor and Shamir [17], and a generalization of this problem called the combinatorial semi-bandit problem with side observations. In the first problem, we have a standard multi-armed bandit, and we observe the reward generated by the arm we pull, as well as rewards generated by some other arms, where the observations are determined by an observation graph chosen by an adversary. In the second problem, we may pull many arms at a time, observe the rewards for all pulled arms, as well as some additional arms, again determined by the adversarially chosen observation graph. The paper develops an algorithm Exp3-IX for the first problem that improves over previous algorithms (it is easier to compute, is anytime, and does not need to know the observation graph before each decision). It then develops another algorithm FPL-IX for the second problem with similar properties.  Both algorithms are provided with regret bounds.Quality:Overall, I think this is a good paper.  The problems studied are interesting and broad, the algorithms are reasonable, and the regret bounds seem well-constructed.  I do have some suggestions to improve the clarity of the paper below, but in terms of quality, I just have two points on which the paper could be improved:1. Even though the algorithms don't need to know G_t in order to make the decision I_t or V_t, they do need to know them in order to update the weights.  This felt hidden to me in the discussion in the introduction, and is an important limitation of the algorithm.2. For the m=1 case, one could also apply FPL-IX.  It would be helpful to provide a discussion of what, if anything, is lost when applying FPL-IX instead of EXP3-IX.Clarity:The paper overall is reasonably clear, but the paper could be made substantially more clear through some additional work to correct two issues:First, there are a number of small errors and omissions listed below. Second, the separate sections of the paper could flow more smoothly together. In reading section 3, I was left wondering why we went to the trouble of developing Exp3-IX in section 2.  While reading section 2, I had the impression that we were developing Exp3-IX in the simple case where we pull one arm per period, and then would generalize in section 3 to the combinatorial case, but then I was surprised when I got to section 3 and found that we were abandoning Exp3-IX in favor of FPL-IX.  I believe that I was given this impression by the first paragraph in section 2, which should more clearly state that Exp3-IX will not be developed further for the combinatorial case, and explain briefly why not.Also, in reading footnote 1 on page 3, and in the discussion of Exp3-IX in previous literature on lines 310-312, it was unclear whether Exp3-IX is being newly proposed in this manuscript, or whether this method already exists in the literature.  Based on examining these papers, it appears that Exp3-IX is new to this manuscript, but still the reader is left to wonder.Similarly, there was some notation defined in section 2 that was redefined in section 3 (\mathcal{F}_t on line 325), and some other notation that was changed or added which could have also been used in section 2 (O_{t,i} on line 326, the vector notation for L and \ell on line 317).Small errors and omissions:1. Reading the proof of Theorem 1, it seems that \ell_t is assumed bounded above by 1.  I assume it is also bounded below by 0.  I did not see this explicitly stated.  While of course this is a standard assumption, it should still be stated.2. The language defining \mathcal{F}_{t-1} on line 149 and (to a lesser extent) line 325 is vague.  I assume that what is meant is that \mathcal{F}_{t-1} is the sigma algebra generated by ( \ell_{s,j}, G_s  : 1\le s < t, (I_s -> i) \in G_s ),i.e., all observable losses and observation structures from interactions up to and including time t-1, but then in a number of conditional expectations in the paper, when conditioning on \mathcal{F}_{t-1} it seems that the paper is additionally conditioning on G_t and \ell_{t,i} for all i, so perhaps \mathcal{F}_{t-1} is meant to include these as well.This specifically seems to be true for equation (2), the unnumbered equations on lines 249, 253, 342, 329, and the statement on line 232 about the dependence of Q_t (Q_t also depends on G_t through o_{t,i}.)Similarly, the conditional expectation in the unnumbered equation on line 165 seems to additionally require conditioning on G_t (and could condition on \ell_{t,i} without affecting the result).If \mathcal{F}_{t-1} is meant to include G_t and \ett_{t,i}, then this should be explained more clearly.  If not, then the conditional expectations should be corrected.Also, the expectation on line 191 should be a conditional expectation.3. \gamma on line 203 should be \gamma_t.4. In Algorithm 1, steps 11 and 23 use the notation E(G_t) while the rest of the paper just uses G_t.  In step 5, why is the 1/d term included, while it isn't on line 214?5. On line 285, this assumption is made for all v \in S.6. On line 334, I believe that the expectation of U_{t,i} should be 1/\gamma_t, not \gamma_t.7. In step 9 of Algorithm 2, the algorithm also requires observing G_t itself, in addition to the observed losses.8. In step 3 of Algorithm 2, I think this should set \hat{L}_0 to 0.9. On line 097, "others" should have an apostrophe.  Also, on line 102, there are parens missing after sqrt{T} and sqrt{dT}.10. line 084, "the losses" is repeated.  Line 085, "thus" should be "this".11. The motivating application could be described more clearly.Significance: This paper represents an improvement over the previously developed EXP3-SET algorithm for the bandit with side-observations problem, both in terms of the convenience of the algorithm, and the quality of the regret bounds obtained.  Also, the second problem studied in section 3 is a significant generalization over previously considered models.Originality: This paper builds on several previous papers, both in terms of the problem studied and the algorithms and analysis developed, but the results presented are novel, and the model is a significant generalization of previously studied models.


Learning Deep Features for Scene Recognition using Places Database
Scene recognition is one of the hallmark tasks of computer vision, allowing definition of a context for object recognition. Whereas the tremendous recent progress in object recognition tasks is due to the availability of large datasets like ImageNet and the rise of Convolutional Neural Networks (CNNs) for learning high-level features, performance at scene recognition has not attained the same level of success. This may be because current deep features trained from ImageNet are not competitive enough for such tasks. Here, we introduce a new scene-centric database called Places with over 7 million labeled pictures of scenes. We propose new methods to compare the density and diversity of image datasets and show that Places is as dense as other scene datasets and has more diversity. Using CNN, we learn deep features for scene recognition tasks, and establish new state-of-the-art results on several scene-centric datasets. A visualization of the CNN layers' responses allows us to show differences in the internal representations of object-centric and scene-centric networks.

This paper introduces a scene-centric database called PLACES with more than 6 million of labeled pictures for scene recognition. The purpose of building such a database is to complement to ImageNet which is an object-centric database, and improve the performance of scene recognition by learning deep features using PLACES. The database was built by scraping search engines with composite queries and utilizing Amazon Mechanical Turk (AMT) for validation. To compare the density and diversity of PLACES with existing databases such as SUN and ImageNet, the authors proposed two metrics and again utilized AMT to get the measures and show the advantages of PLACES. The authors also compared the features learned from PLACES and from ImageNet, and showed the two databases are complementary to each other. In particular, PLACES can result in to better performance for scene recognition and ImageNet better for object recognition.Strong points of this this work:1. A solid work on dataset building.2. Detailed and convincing better result than existing scene databases. Shows the power of data.Minor concerns and comments:1. Limited algorithm contribution.2. For density, nearest neighbors are obtained based on the GIST feature. Though this setting is the same for all the databases, I wonder if depending on a particular feature will lead to any bias for the density metric.3. For diversity, judgment can be quite difficult if judges are presented with two random images, as it doesn’t make sense to judge the similarity between two unrelated random images. I wonder if the authors have similar observations in their AMT experiment.4. As PLACES and ImageNet are two complementary databases, how about to mix them together to train a model that is good for both object and scene recognition?


Parallel Sampling of HDPs using Sub-Cluster Splits
We develop a sampling technique for Hierarchical Dirichlet process models. The parallel algorithm builds upon [1] by proposing large split and merge moves based on learned sub-clusters. The additional global split and merge moves drastically improve convergence in the experimental results. Furthermore, we discover that cross-validation techniques do not adequately determine convergence, and that previous sampling methods converge slower than were previously expected.

Summary: The paper develops a MCMC algorithm for the Hierarchical Dirichlet process (HDP) that accommodates splits and merges and can be parallelized. Issues with respect to developing effective splits are addressed and controlled experiments demonstrating the algorithm's effectiveness are presented.This is a well written, technically sound paper. The primary contribution lies in extending the DP sub cluster framework of [1] to HDPs. Although, this is somewhat incremental, the innovations required to transition the ideas in [1] to HDP inference are significant enough to warrant publication. Detailed Comments:1) Global split/merge moves: I like these moves and it is certainly believable that to escape certain "local optima" moves that distribute a topic's mass globally are required. However, given the significant extra cost of having to loop through all data instances it is important to clearly demonstrate the benefit of incorporating such moves. To this end, the experiments with and without the global moves on the toy bars dataset are not convincing. The authors do mention that they notice significant improvements on the larger real world datasets. I would like to see an additional plot added to Figures 4 and 5 from a handicapped version of the algorithm that uses only local moves.2) It appears that at each iteration, the topic to be globally split is being chosen uniformly at random. Perhaps a more data driven proposal could be utilized here. For example, the negative cross entropy measure (C) defined in lines 375-376 provides a nice heuristic for determining "confused" topics. A topic, many of whose words have high probability under other topics, could be  selected with higher probability for a split. 3) The issue of approximating the Hastings ratio for the merge moves is somewhat confusing. Are the MCMC guarantees maintained in spite of the approximation? 4) Is there a reason no experimental comparison against Wang and Blei's SAMS sampler is presented? The code is publicly available and should really be compared against.


Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets
To cope with the high level of ambiguity faced in domains such as Computer Vision or Natural Language processing, robust prediction methods often search for a diverse set of high-quality candidate solutions or proposals. In structured prediction problems, this becomes a daunting task, as the solution space (image labelings, sentence parses, etc.) is exponentially large. We study greedy algorithms for finding a diverse subset of solutions in structured-output spaces by drawing new connections between submodular functions over combinatorial item sets and High-Order Potentials (HOPs) studied for graphical models. Specifically, we show via examples that when marginal gains of submodular diversity functions allow structured representations, this enables efficient (sub-linear time) approximate maximization by reducing the greedy augmentation step to inference in a factor graph with appropriately constructed HOPs. We discuss benefits, tradeoffs, and show that our constructions lead to significantly better proposals.

The paper proposes to greedily find a series of M diverse solutions to a structured output problem. First, the structured output prediction problem is expressed in terms of MAP inference in a factor graph. Next, this graph is augmented by a higher-order factor that encourages diversity from previously found solutions.  Repeated MAP inference in this augmented factor graph (with iteratively updated higher-order potentials) then gives a series of M diverse solutions. Different task-specific user-defined diversity measures lead to inference problems that are a special case of either label costs, co-operative cuts or cardinality potentials inference. The experiments show similar performance to Diverse M-Best [1, cited] on two datasets for one proposed diversity measure whereas the other proposed diversity measures perform significantly worse than [1] in general, but have some advantages on single instances of the dataset.Quality: The paper is well written (apart from some minor notational issues raised below) and the structure is mostly clear. It offers some theoretical insights in the DivMBest method and extends that model to higher-order diversity potentials. The paper contains most relevant proofs.The proposed diversity measures, however, seem to lack some desirable properties:* The diversity measures in Eqs. (5) and (6) are invariant to label flips. For instance, if in one segmentation, two objects are labeled as cat and dog, then there is no diversity-related reward for flipping those two labels, i.e. labeling the objects as dog and cat, respectively. * Only new labels are rewarded, structure (location and size of the superpixels,...) is not taken into account, e.g. labeling a different object as dog, or increasing the size of the dog object.* The greedy fashion of the algorithm has drawbacks when the diversity measure from Eq. (5) is utilized: For instance, if in Fig. 2a the MAP solution y_1 yields a segmentation containing three (super)pixels with a dog, person and horse, then there is no further marginal gain for diversity. Also, the general diversity in Eq. (6) will favor segmentations with many distinct labels. Diverse solutions may, however, also contain fewer distinct labels.* The parsimony factor raises some concerns:  i) The parsimony factor should naturally be included into the diversity term, not the relevance term (l. 227) since the learned weights in the relevance term should be independent of the diversity term. If a diversity term without bias towards more labels is designed, the parsimony factor would be unnecessary. In other words, if the MAP solution of the factor graph without diversity term is to be found (i.e. minimization of the relevance term), the parsimony factor should not be included.  ii) It is currently chosen in a rather heuristic manner (e.g. how to choose/learn c(l) in line 248?).   iii) It is unclear whether/how this term is weighted against the relevance AND/OR diversity term: In l. 252 it is assigned the same weight as the diversity term, whereas this lambda is missing in l. 396.   iv) Both background-only segmentations and a segmentation containing only new labels are rewarded by zero (using l. 252). Hence, many segmentations will only consist of background, such as in Fig. 3.   v) Another major concern is that the sum in l. 252 may become negative, and consequently the requirements for the objective in Eq. (2) are violated.Furthermore, the paper does not elaborate on how the maximization in Eq. (2) is done over V\S^(t-1). If the maximization is done over V instead, it is not clear how it can be ensured that an optimal solution a^t is chosen which is not already contained in S^(t-1).The concerns above are mirrored by the experiments which perform significantly worse than the competitive method [1]. The Hamming balls potentials address some of these issues and achieve competitive results, but approximation errors from maximizing the loose lower bound (eq. 10) are included. Why is it not possible to include the exact marginal gain into the higher-order potential? Moreover, the Hamming ball section would profit from some reformulations and derivations, since this -- according to the experimental results -- seems to be the only useful diversity measure proposed in this paper.The experimental section would benefit from a more thorough analysis: * It would be interesting to see how robust the results are with different diversity parameters.* A combination of DivMBest and the proposed method, i.e. including the unary potentials from DivMBest AND the higher-order potentials, may perform much better.* It is not stated how the parameters are chosen for the experiment in 5.2, how the parsimony factor weight is chosen, and how the Hamming ball radius k chosen.* It is not clear why the MAP solutions in Tab. 2 are different.* Why is only recall reported?While the authors claim "significantly higher oracle accuracy" (l. 94), according to the Results Section the method performs significantly worse than DivMBest in all cases except for the Hamming ball measure (which they state is rather similar to DivMBest), where results are comparable.  Clarity:* Fig. 2a (not referenced in the text) shows 6 examples which seem to be diverse segmentations. With the proposed count diversity, however, the greedy algorithm will only reward diversity for the first 1-3 segmentations (dependent on the order). Hence, this example should only include 1-3 different segmentations.* The notation in Eq. (9b), | B | and | union(B) |, where B are balls, is unclear.* The derivation of the equation in l. 287 is unclear and so is Fig. 2b: In Fig. 2b, some balls seem to have b=9, some b=7,...; |S|=4. It is not intuitively clear, why b/|S| is used. This makes it difficult to understand the exact connection to DivMBest [1]. The supplementary material should be extended appropriately.* I_y in ll. 289 and 307 does not seem to be the same.* Suppl. Fig 4 bottom is confusing: How can y_2 (Hamming) be black completely and y_3 contain a label? y_2 shouldn't have changed the objective, right?* l. 98: [14] do not learn a MIXTURE of M models* It is not mentioned in Sec. 3 that r(y) is required to be submodular.* The relevance function must be non-negative, as stated in l. 160 and enforced by the constraint in Suppl. l. 58. I assume the constraint is added in a cutting-plane fashion, since it involves all y \in Y. In practice, this constraint may be violated for some y \in Y? What is the procedure then?Originality and Significance:The derivation of diverse M-best solutions using set functions and maximizing their marginal gains is novel and sheds new light on DivMBest [1] for this specific model class. Extending [1] to higher-order potentials seems promising, but experimental results show only incremental and partial improvement compared to [1]. Minor comments:Equations 4, 5, 8, 9a: Instead of { G_i | ... } these sets should be over indices of groups { i | ... } rather than over one group G_i157: r(y) non-negative only ensures that r(.) is monotone.Fig. 2 is not referenced in the text19: is a challenging in structured prediction problems -> is a challenging task in structured prediction problems41: solve [...] problems that often fairly ambiguous -> solve [...] problems that often are fairly ambiguous81: where [k] is... -> leave out, since it is defined by [L] = {1,...,L} already143: there is broad class -> there is a broad class146: use structured a relevance function -> use a structured relevance function156: and a feature vectors -> and feature vectors157: It must be stressed that r(.) is only monotone if r(.) is linear and all summands are non-negative. 239: diverity -> diversity245: missing brackets around the differenceSupp. 66 and below: w* -> \hat w*266: leave out "large"289: I(y') -> I(y)292: presentend -> presented-----------I have increased the quality score by 2, assuming the authors will make all promised additions to the camera-ready version.


Consistency of Spectral Partitioning of Uniform Hypergraphs under Planted Partition Model
Spectral graph partitioning methods have received significant attention from both practitioners and theorists in computer science. Some notable studies have been carried out regarding the behavior of these methods for infinitely large sample size (von Luxburg et al., 2008;Rohe et al., 2011), which provide sufficient confidence to practitioners about the effectiveness of these methods. On the other hand, recent developments in computer vision have led to a plethora of applications, where the model deals with multi-way affinity relations and can be posed as uniform hypergraphs. In this paper, we view these models as random m-uniform hypergraphs and establish the consistency of spectral algorithm in this general setting. We develop a planted partition model or stochastic blockmodel for such problems using higher order tensors, present a spectral technique suited for the purpose and study its large sample behavior. The analysis reveals that the algorithm is consistent for m-uniform hypergraphs for larger values of m, and also the rate of convergence improves for increasing m. Our result provides the first theoretical evidence that establishes the importance of m-way affinities.

In this paper the authors present theoretical guarantees for the accuracy of a spectral partitioning algorithm on uniform hypergraphs as well as experiments which support their claims. The proofs substantially build on prior work [5] regarding estimating misclustering.This is an interesting paper and quite clearly written. The spectral clustering method presented and the theoretical guarantees appear novel. However, there are some questions in the minds of this reviewer which prevents declearing it a very good paper. Hopefully a revision that answers these issues and improves the presentation will follow.1. Spectral partitioning for regular graphs that use eigenvectors of a Laplacian matrix formed from the adjacency matrix is well established. Applying a spectral technique for (uniform) hypergraphs got less attention, but there is prior art as also cited in the paper (especially [8]); these seem to all come from image processing backgrounds. There exists prior work from other areas that similarly forms a Laplacian (like) matrix derived from the hypergraph and uses eigenvectors of this matrix to perform partitioning or clustering. (Two examples from different fields: Multilevel Spectral Hypergraph Partitioning with Arbitrary Vertex Sizes J. Y. Zien, M. D. F. Schlag, and P. K. Chan in IEEE trans. on computer aided design of integrated circuits and systems vol 8, no 9 or Laplacian eigenvalues and partition problems in hypergraphs J. A. Rodriguez in Applied Mathematics Letters, volume 22, issue 6). One wonders how the proposed/discussed method differs from those other approaches.The authors could properly place their work in the context of spectral techniques for hypergraphs in general and not just geometric groupings. It seems to this reviewer that the use of the flattened adjacency tensor is fairly new and therefore improvements to it and theoretical results might be novel. The authors could make it more clear what is novel and what is reproduced from [8] regarding the algorithm in their discussion.2. The algorithm described in [2] slightly differs from algorithm 1 in that a normalization step is missing prior to the k-means clustering of the rows of X_n.This reviewer asks the authors to discuss (in the paper and outside) what is the source of the difference between Algorithm 1 and that of [2]. Do normalization take place in the implementation? 3. In [5] on page 15 it is noted that the eigenvectors belonging to the largest absolute eigenvalues of L are (and should be) used in their framework. Since the proofs rely on [5], this reviewer asks the authors to discuss the form of the Lagrangian used and the subsequent eigenvector selection and how the misclustering estimates are affected. (See theorem 3.1 of [5] as well.)4. Also in [5] on page 18 it is noted that their theorem 3.1 "depends on the true optimum of the k-means function", or more precisely a certain inequality holding. The authors of [5] repeated the k-means step with various random initializations as part of their simulations until the inequality did hold. This aspect should be addressed by the authors regarding their theoretical guarantees and experiments.Some additional comments:While the paper presents a spectral clustering algorithm it could be still beneficial to mention other state of the art hypergraph clustering approaches, for example Metis.Throughout in the paper the nottaion is using n as subscript to remind the problem size. This only seems to make the nottaions slightly more complicated and this reviewer feels that the n subscript could be safely dropped. The notation can even be confusing, for example C_n is a kxk matrix and it is not denoting an nth iterate.Line 104: "expected quantities of entries of A" - this does not sound right, this reviewer cannot parse "quantities of entries"Throughout the paper there are mismatched articles like: "an hyperedge", "a m-uniform" etc.Line 192: It seems that C_n is positive semidefinite and that could be mentioned.Line 200: Some intuition could be given as to the various conditions.Line 206: The statement |M| = O(T) almost surely may be confusing a bit at first for some readers. (It was for this reviewer.) This is interpreted as there exist a k such that |M| <= kT with probability 1. Is that correct?Line 382: should be: "number of problems" AFTER AUTHOR FEEDBACK:I mentioned above the issue of the selection of the k eigenvectors of the Laplacian. I understand and thank the author's response regarding the Laplacian formulation. I am still a bit confused what formulation is used.To elaborate: In [5] (on which this paper and the results build) the authors emphasize that the eigenvalues need to be ordered by absolute values. Now the standard definition of the Laplacian implies that it is positive semidefinite and so taking the abolute values appears unnecessary. However the authors of [5] are adamant; a quote from [5]:"The above theorem suggests that before finding the largesteigenvalues, you should first order them by absolute value. This allows forlarge and negative eigenvalues. In fact, eigenvectors of L corresponding toeigenvalues close to negative one (all eigenvalues of L are in [−1, 1]) discover“heterophilic” structure in the network that can be useful for clustering."(see page 16 in the discussion of theorem 3.1)I repeat: this is mostly concerning since the submitted paper builds on and cites [5] heavily so this reviewer is puzzled: Did the authors of [5] make a mistake regarding the absolute value? Or did they use a different Laplacian than the submitted paper?I do not generally hold the authors of the submitted paper responsible for the correctness of a cited work, but to the extent that no errors are propagated.


The Blinded Bandit: Learning with Adaptive Feedback Elad Hazan Technion
We study an online learning setting where the player is temporarily deprived of feedback each time it switches to a different action. Such model of adaptive feedback naturally occurs in scenarios where the environment reacts to the player's actions and requires some time to recover and stabilize after the algorithm switches actions. This motivates a variant of the multi-armed bandit problem, which we call the blinded multi-armed bandit, in which no feedback is given to the algorithm whenever it switches arms. We develop efficient online learning algorithms for this problem and prove that they guarantee the same asymptotic regret as the optimal algorithms for the standard multi-armed bandit problem. This result stands in stark contrast to another recent result, which states that adding a switching cost to the standard multi-armed bandit makes it substantially harder to learn, and provides a direct comparison of how feedback and loss contribute to the difficulty of an online learning problem. We also extend our results to the general prediction framework of bandit linear optimization, again attaining near-optimal regret bounds.

The paper considers an online learning game with a special sort of partial information, where the learner only gets bandit feedback on its loss whenever it sticks with its previously chosen action. The authors show the surprising fact that in this so-called blinded bandit setting, it is still possible to attain regret guarantees of the same order as in the standard multi-armed bandit setting by using a simple modification of the classic Exp3 strategy. The results are also extended to the case of bandit linear optimization.Originality & Significance--------------------------In light of recent results by Dekel et al., it is indeed surprising that this learning setting is no more difficult than the standard bandit setting. Even though I wouldn't call this observation groundbreaking, it still constitutes an interesting contribution to our general understanding of the role of information in online learning. The algorithm itself and its analysis are quite elegant: the approach is based on a simple and clever subsampling strategy of the stream of observations that effectively separates in time the selection of actions and observing the associated losses. While specifically tailored to this application, this strategy can be possibly applied in other online learning problems with delayed feedback.Quality & Clarity-----------------The paper is written very well and the technical quality is very good, apart from a few minor technical issues. Precisely, I have located one bit in the analysis that needs to be fixed.Expectations and conditional expectations are treated quite liberally throughout the paper. The first occurrence of this problem is to be found near line 264, where the authors use the fact that "b_{t+1}" is independent of l_t(x_t)(1-b_t)", which of course only holds when conditioned on the history up to time t. While this specific problem doesn't stir up any trouble, the same bug shows up on line 274, where the distributions of x_t and x_{t-1} are p_{t} and p_{t-1}, respectively, but under different conditions. Treating this issue complicates the analysis a bit (by an additional application of the tower rule and Jensen's inequality), but it seems that the analysis still goes through -- although the authors should provide a precise proof before the paper can be accepted.Detailed comments-----------------065: The example in this paragraph, while sensible, is probably not the best fit for this framework, as the increased activity *does* influence the true rewards, not only their observations -- the two are closely tied together. Either remove this example or elaborate if you still think it's correct.169: "maintains a probability" -> "maintains a probability distribution"172: p_1 -> p_t297: "The claim is trivial for rounds in which no update occurred" -- This statement is difficult to interpret, as the claim is about an expectation that integrates over the randomness inducing the updates. That is, it is meaningless to refer to action switches unless one conditions on the sequence of b_t's. Please fix this.388: Note that the last inequality follows from n \ge 2.332: The result of Dani et al. is proven for continuous decision spaces by laying an appropriately-sized grid on the decision space and running GeometricHedge on the resulting discrete action set. That is, their algorithm does not match the one presented as Algorithm 2. If you can prove Lemma 4 for this variant of the algorithm, please include it, otherwise, state that the decision space should be finite. In the latter case, the integrals in the proof of Lemma 5 should be replaced by sums.


Sequential Monte Carlo for Graphical Models
We propose a new framework for how to use sequential Monte Carlo (SMC) algorithms for inference in probabilistic graphical models (PGM). Via a sequential decomposition of the PGM we find a sequence of auxiliary distributions defined on a monotonically increasing sequence of probability spaces. By targeting these auxiliary distributions using SMC we are able to approximate the full joint distribution defined by the PGM. One of the key merits of the SMC sampler is that it provides an unbiased estimate of the partition function of the model. We also show how it can be used within a particle Markov chain Monte Carlo framework in order to construct high-dimensional block-sampling algorithms for general PGMs.

The authors propose a method for applying sequential Monte Carlo (SMC) inference to probabilistic graphical models (PGMs). Specifically, the authors show how SMC can be applied to infer latent variables and estimate the partition function (model evidence, posterior normalisation) when a PGM is represented as a factor graph. The authors also show how SMC inference for PGMs can be used as a kernel in an MCMC sampler (via PMCMC) to perform block sampling of latent variables. Finally, the authors apply their method to several datasets showing that their method can i) accurately estimate the partition function and ii) PMCMC sampling can be used to introduce block resampling moves of latent variables with very low auto-correlation.Using SMC for inference in PGMs is an interesting idea that exploits the fact that SMC inference can be done on fairly arbitrary sequence of distributions, provided that the final distribution in the sequence is the desired target distribution. The authors clearly explain how the factor graph representation of a PGM can be decomposed to define such a sequence of distributions. I found the review of factor graphs well written but lacking in references to more detailed resources. I felt a similar section reviewing SMC inference, independent of the application to PGMs would greatly aid paper. As the paper is written it may be difficult for readers without a fairly detailed knowledge of the SMC literature to access the paper. For example the authors talk about adjustment multipliers and fully adapted samplers in section 3.2. While these are important concepts for building efficient SMC samplers, they are also potentially unnecessary complications when presenting the basic idea of SMC inference in PGMs.The experiment in section 5.1 is sound and demonstrates that the estimates of the partition function generated by SMC are comparable to state of the art methods. I am not satisfied with the results of section 5.2. I found it surprising that the authors were able to include $\theta$ as part of the SMC chain, particularly that the variable would be added first during inference. For the synthetic dataset (Figure 6a) the authors show that the SMC algorithm performs reasonably well. I suspect this is due to the small number of topics and the corresponding small dimension of $\theta$. It would be useful to repeat the experiment with a larger number of topics so that the dimension of $\theta$ is larger and see if performance remains competitive with LRS. It is also not clear how the authors are determining the performance on the real data  (Figures 6b-c); are they comparing against LRS or looking at the variability of estimates?The experiment in section 5.3 explores the use of PMCMC for PGMs. Building PMCMC samplers that can update blocks of latent variables by SMC and parameters by Metropolis-Hastings or Gibss sampling could be very powerful. Unfortunately, I feel the experiment is incomplete. The authors only examine auto-correlation times (Figure 7), but do not show a comparison of time complexity of the iterations for each method, or how accurate the different inference schemes are for a fixed computational budget.A few other small issues- For figures 3 and 6a I am not sure what N on the x-axis refers to.- The sentence "Both methods converges to the true value, however, despite the fact that it uses much fewer computational SMC performs significantly better in terms of estimator variance." on lines 369-371 is grammatically incorrect and confusing.- In supplemental section 1.1 I couldn't find the definition for the functions $\kappa$ or $\mu$. They are defined implicitly as the mean and dispersion of a Von Mise distribution, but it would be useful to explicitly state what they are.


Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
We present a policy search method that uses iteratively refitted local linear models to optimize trajectory distributions for large, continuous problems. These trajectory distributions can be used within the framework of guided policy search to learn policies with an arbitrary parameterization. Our method fits time-varying linear dynamics models to speed up learning, but does not rely on learning a global model, which can be difficult when the dynamics are complex and discontinuous. We show that this hybrid approach requires many fewer samples than model-free methods, and can handle complex, nonsmooth dynamics that can pose a challenge for model-based techniques. We present experiments showing that our method can be used to learn complex neural network policies that successfully execute simulated robotic manipulation tasks in partially observed environments with numerous contact discontinuities and underactuation.

Summary:The authors propose the combined use of trajectory optimization and policy learning for systems with unknown dynamics approximated by linear local models. The proposed trajectory optimization algorithm is based on DDP/iLQG methods. The local linear models are assumed Gaussian and obtained from samples returned by the previous policy solution. The paper introduces the iLQG-based method such that the approximated local models can be used. Optimization with approximated models is made possible by limiting the changes of the modified iLQG solution using KL-divergence as a constraint. The paper also presents the use of their trajectory optimization method within the framework of Guided Policy Search previously proposed in [11] such that richer parameterized policies can be learned by iteratively swapping between the trajectory optimization and the policy learning steps. Results are evaluated in simulation.Quality:The paper is of high quality. It is clearly written. Sometimes the paper relies too much on prose (see comments below) and support for a few statements are not clearly found. The paper is of high interest for the robotics and robot learning community especially due to the "unknown dynamics" feature, however, large part of the merit is attributed to the fact that it builds upon the the contributions of [11, 12, 13]. The evaluation of the method is of high quality and extensive and several relevant state-of-the-art methods are selected for comparison. The simulated tasks are interesting and show in a clear way the differences in performance. Clarity and Questions:Certain parts of the paper are not clear and although he supporting information may be somewhere in the paper it is not always easy to find.. Please, define the use of "unknown dynamics" and "hybrid" right at the introduction. Unknown dynamics inevitably induces the "model-free" image, but the paper does not seem to consider itself as a model-free method. At the same time, the idea of creating a model from samples is usually regarded as model-based (such as PILCO). I personally would not introduce the new term "unknown dynamics" but rather use something like "local Gaussian models" instead.. In line 52, "parameterized policy never needs to be executed on the real system" needs clarification. What I understand from this  sentence is that you would use the Gaussian controller during the whole learning process, and only at the end learn the parameterized policy based on what the Gaussian controller found (?). In that case it seems that you are not using the GPS framework as the parameterized policy is not being influenced by the trajectory optimization and vice-versa.. Line 59, the conclusion "...transforms a discontinuous deterministic problem into a smooth stochastic one" is very strong but the intuition that lead to it is not convincing. Is there a better way to explain it? A related request: since simulations with contact are difficult to model and local models do not capture discontinuity the paper would become much stronger with experimental results on a tasks with real physical contact or impact. . Although the GMM for background dynamics distribution seems to be very beneficial (in lines 400-401 it even seems to be intrinsic part of the method), it is only in Section 3.2 that GMMs are firstly mentioned. Until this point the whole method is explained as if there is no GMM. This can be a little confusing as at this point I do not know if GMM is part of the method or GMM is an extra feature that improves the method. As it seems to be always a good thing, why not present the method as unknown dynamics approximated by a mixture of Gaussians?. How does the system behave under saturation? Any real physical robot will have actuation limits. Does it compromise the induction of Gaussian trajectory distributions if the commands from the controller are saturated?. The fact that the iLQG with a true model performs worse than your method (which builds on iLQG but uses approximated linear models) is counter-intuitive. How it is possible?Originality and Significance:The paper is incremental in relation to the works of [11,12,13] although the contribution it adds is significant. The idea of being able to optimize trajectories and design a controller for an unknown system dynamics is of high interest for the robotics community, which was not previously addressed by [11,12,13].


Permutation Diffusion Maps (PDM) with Application to the Image Association Problem in Computer Vision
Consistently matching keypoints across images, and the related problem of finding clusters of nearby images, are critical components of various tasks in Computer Vision, including Structure from Motion (SfM). Unfortunately, occlusion and large repetitive structures tend to mislead most currently used matching algorithms, leading to characteristic pathologies in the final output. In this paper we introduce a new method, Permutations Diffusion Maps (PDM), to solve the matching problem, as well as a related new affinity measure, derived using ideas from harmonic analysis on the symmetric group. We show that just by using it as a preprocessing step to existing SfM pipelines, PDM can greatly improve reconstruction quality on difficult datasets.

This paper applies the diffusion map framework in a permutation setting similar to the cited work [23] which works with rotations in primarily image association problems. The goal is to develop a global association scheme that can better handle the presence of duplicate structures. The authors support their proposed method with experiments on various real image datasets.This is a clearly written paper and the reviewer expect it to be interesting to the NIPS audience. This reviewer sees the theoretical contribution as the idea of viewing the transformations that need to be coherent as a group and the observation that a preprocessing can bring benefit to existing intricate algorithms and implementations.However there is a concern also in this regard. One may argue that the idea of thinking in terms of an algebraic group is essentially present already in the cited [23]. The authors describe the case when the transformations are matching and can be viewed as permutations, they derive the formulas but also point out that they are very similar to those of in the rotation setting.As a result this reviewer is somewhat undecided, but the strong experimental section tipped the balance in favor of recommending acceptance.A few detailed comments:The authors should consider citing: Coifman, R.R.; S. Lafon. (2006). "Diffusion maps". Applied and Computational Harmonic Analysis, that introduced diffusion maps.Line 135: typo in x_p^i \sim x_q^jLine 150: it may make sense that at the first use of "group" the authors would use "algebraic group" considering the audience and venue (not a group theory conference)Line 209-210: The Hermitian conjugate is traditionally denoted by M^*. This reviewer sees no reason to deviate from this custom. Also note the transpose is typically marked by capital T, incorrect in (1).Line 211: Define compact group. Why we need compact?


Learning Time-Varying Coverage Functions
Coverage functions are an important class of discrete functions that capture the law of diminishing returns arising naturally from applications in social network analysis, machine learning, and algorithmic game theory. In this paper, we propose a new problem of learning time-varying coverage functions, and develop a novel parametrization of these functions using random features. Based on the connection between time-varying coverage functions and counting processes, we also propose an efficient parameter learning algorithm based on likelihood maximization, and provide a sample complexity analysis. We applied our algorithm to the influence function estimation problem in information diffusion in social networks, and show that with few assumptions about the diffusion processes, our algorithm is able to estimate influence significantly more accurately than existing approaches on both synthetic and real world data.

This paper defines temporal coverage functions, a generalization of coverage functions with dependence. It introduces a method to learn them from previous time history. It also includes experimental validation on real data sets.The paper is well written, and clear. It is also very technical, and was not an easy read for me. Judging its quality is even harder. I think it would help if the authors could demonstrate with an example what a time-varying coverage function means in practice, for a given data set. That would help me refine the quality score I given in my review.It is not clear to me which of the baseline algorithms in section 7.1 were also using time information, so it would be great it the authors could clarify this? I believe that for a fair comparison the baseline algorithm should use time information as well.


Convex Deep Learning via Normalized Kernels
Deep learning has been a long standing pursuit in machine learning, which until recently was hampered by unreliable training methods before the discovery of improved heuristics for embedded layer training. A complementary research strategy is to develop alternative modeling architectures that admit efficient training methods while expanding the range of representable structures toward deep models. In this paper, we develop a new architecture for nested nonlinearities that allows arbitrarily deep compositions to be trained to global optimality. The approach admits both parametric and nonparametric forms through the use of normalized kernels to represent each latent layer. The outcome is a fully convex formulation that is able to capture compositions of trainable nonlinear layers to arbitrary depth.

The submission describes a convex deep learning formulation that leverages a number of key ideas.  First, a training objective is proposed that explicitly includes the outputs of hidden layers as variables to be inferred via optimization.  These are linked to linear responses via a loss function, and the net objective is the sum of these loss functions across the layers, plus some regularization terms.  Next, a number of changes of variables are performed in order to reparameterize the objective into a convex form, heavily leveraging the representer theorem and the idea of value regularization.  We are left with a convex objective in terms of three different matrices (per layer) to optimize.  In particular, one of these matrices is a nonparametric ‘normalized output kernel’ matrix, which takes the place of optimizing over the hidden layer outputs directly; however, this leads to a transductive method where we must simultaneously solve the optimization for training and test inputs.  Finally, a relaxation must be performed in order to obtain a truly convex formulation, since the set of valid kernel matrices is generally non convex (possibly discrete, given assumptions on the hidden layer values).  Experiments are shown comparing the method to a previous two-layer convex model on small datasets.Quality:The basic ideas presented seem technically sound, although I was unable to rigorously verify many aspects of the derivation.  It is no small technical feat to achieve a convex deep learning formulation, and the authors make use of several very sophisticated tricks, gyrations, assumptions, and clever observations to achieve this goal.  Most of the derivation of the method consists of reparameterizations of the objective that leverage the representer theorem.However impressive this technical feat may be, my overall impression is that the resulting method is more of an academic curiosity than a practical method, for a few reasons.  First, the derivation of the method makes a few critical assumptions and relaxations to achieve a convex objective, such that we are left wondering whether the resulting model really captures the essence of a traditional deep network.  The most significant of these is relaxing the set of output kernels to a convex set.  The original motivation for the objective seemed to be a Boltzmann-machine-style model, where the hidden units are restricted to take on binary values--except the submission proposes to infer the hidden states deterministically.  The convex relaxation of the output kernel then effectively relaxes this binary constraint on the hidden units, which brings us a little farther away from the original motivation, and raises doubts as to how the new objective might be interpreted.  A similar argument can be made for the loss, which seems to replace a saturating loss with a hinge loss.  The submission gives little intuition as to why this is justified.  The transductive nature of the algorithm is also disappointing, and doesn’t seem scalable to large datasets.Unfortunately, I found several important steps in the derivation to be beyond my ability to comprehend.  However, I have at least one technical doubt that the authors may want to address—the reparameterizations of the objective induce constraints on the matrices that are not obviously convex constraints: for example, S \in MRK in Eq. 8, where M and K are in convex sets, and R is an appropriate matrix.  It is not clear to me whether this set is convex.  Perhaps the authors could provide a reference.  Also, I think \Phi U in Eq. 5 should be U \Phi.The experiments are very weak and add to my doubts regarding the practicality of the method.  The datasets used are small subsets of the original sets, leading me to believe that scalability is indeed an issue.  The only comparison is to a two-layer version of the method.  An obvious missing comparison is to a traditional deep net.  For that matter, even a comparison to a shallow method should at least be included as a baseline.  Also, I would be interested to see how nonlinear optimization of the original non convex objective fares compared to the convex version.Clarity:A lack of clarity is perhaps the weakest point of the submission.  The authors seem to have in mind a particular mental model and rationalization for their approach that does not come through in the submission.  For example, in several places, passing remarks are made about the nature of the loss function (e.g., convexity in one slot or the other), but no such assumptions are clearly made up-front.  It would also very much help to clearly delineate the connection and inspiration from neural-net-like Bayes nets or Boltzmann machines in order better motivate the objective.  For example, what might the loss look like concretely for a Bayes net, and how does this compare to the actual loss used?  What effects does the difference have on the expressive capacity of the method?  The submission should makes these issues more clear.I also found several critical points in the derivation to be lacking any kind of meaningful explanation.  This is especially true in the vicinity of Eq. 5-7.  This is the core of the method, but very little explanation is provided besides a few references to value regularization and some recent papers on clustering.  This is very disappointing, because I found section 2 to be fairly readable in comparison, but that section essentially ends in a dead-end.  I would have preferred it if section 2 were much shorter, and section 3 were longer.Originality:Although this work seems to be inspired by earlier work on two-level convex networks, and is based on the same basic model, the trick to generalize to arbitrary nesting seems novel enough.  The proposed optimization algorithm also seems novel.Significance:As the paper itself acknowledges, obtaining a convex training formulation is not a universally accepted aim of deep learning researchers, nor do local minima seem to be a serious problem in real networks.  The proposed training procedure is fairly complicated and seems to scale poorly, judging by the experiments.  The formulation is fairly inflexible, working only with a narrowly defined set of loss functions, and the resulting algorithm is transductive, which further hurts performance and ultimately limits scalability.  In the end, even though the optimization is convex, this is only obtained by making a critical relaxation that limits our ability to constrain the domain of the hidden layer outputs.  For all these reasons, the immediate practical impact of this work is likely to be very low.On the other hand, it is possible that this work could stimulate further work on the subject, since the very idea that a convex formulation might capture the important aspects of a deep network is interesting by itself.  Future work might elucidate the intrinsic theoretical limitations of this formulation versus traditional deep nets.


Efficient Minimax Signal Detection on Graphs
Several problems such as network intrusion, community detection, and disease outbreak can be described by observations attributed to nodes or edges of a graph. In these applications presence of intrusion, community or disease outbreak is characterized by novel observations on some unknown connected subgraph. These problems can be formulated in terms of optimization of suitable objectives on connected subgraphs, a problem which is generally computationally difficult. We overcome the combinatorics of connectivity by embedding connected subgraphs into linear matrix inequalities (LMI). Computationally efficient tests are then realized by optimizing convex objective functions subject to these LMI constraints. We prove, by means of a novel Euclidean embedding argument, that our tests are minimax optimal for exponential family of distributions on 1-D and 2-D lattices. We show that internal conductance of the connected subgraph family plays a fundamental role in characterizing detectability.

** I have read the authors' response and the other reviews. The authors have provided satisfactory answers to my concerns. I agree with one of the other reviewers that the presentation is dense, but I think the authors have done a good job considering the space limitations. It would certainly be a better paper with no page limit, but the authors don't have that option. The experimental performance analysis could be improved, but considering the strength of the theoretical contribution, the paper remains for me in the top 15-25 percent of accepted NIPS papers. **Summary of the paper: The paper addresses the problem of signal detection for the case where the signals reside on connected subgraphs of a graph that models the relationships between a set of entities. The authors discuss how the problems can be formulated as optimization of objective functions defined on the subgraphs. A straightforward search over the subgraphs is computationally infeasible, so the authors present a highly novel approach that leads to computationally efficient tests. The paper includes proofs that the tests are nearly minimax optimal for the exponential family of distributions and graphs satisfying the polynomial growth property. The paper concludes with an analysis of synthetic and real datasets.Strengths:(1) The paper addresses a problem of growing importance and presents novel approaches for statistical tests. These are computationally feasible and the authors have managed to theoretically characterize the performance. The paper is, for the most part, very well-written, so that the concepts and the development of the approach is clear and easy to follow. The proofs are not straightforward and involve techniques that are of interest in their own right. Weaknesses:(1) The presented proofs (in the supplementary material) for two of the key results are limited to the 2D-lattice case, but are claimed in the paper for more general settings. The authors argue in each case that this is “For simplicity”, but I don’t think the extension to the claimed polynomial growth is completely obvious, and I’d prefer to see a “For completeness” proof follow the simple arguments. There is also a bold claim that the “analysis extends beyond Poisson & Gaussian models and applies to general graph structures and models” but there is little support for this claim. While the few sentences of argument are reasonable and it is likely that the analysis can be extended as the authors claim, the presented material does not genuinely support the claim.  (2) More detail could be provided for the simulation comparison. Apologies if I missed them, but what are the ground-truth subsets for the lattice examples? Are these constrained to be different from the completely arbitrary connected subsets considered by the SA algorithm? Are they randomized? Why not show a comparison for several choices of \gamma rather than just a value corresponding to thin sets? Other comments:The sketch of the proof of Theorem 3 is hard to follow. I think this largely arose for me because I didn’t understand what was meant by “M is connected”. M is a matrix. The only graph associated with it previously in the text is H, so I initially read this as “H is connected”, which means that the argument is circular. After a careful reading of the actual proof, I understand that what is meant by “M is connected” is that the graph with nodes Supp{Diag(M)} and edge set Supp(M) is connected.In the supplementary material, proof of Theorem 3, I think it is worth adding a sentence prior to (16) explaining that, due to the presence of the anchor node implied by M, the disconnected nature of H can only arise from A, the edge set of the underlying graph. This logical step wasn’t obvious to me at first.Theorem 3 proof: Line 3: |\overbar{C}| = k_2The connection between the proof of Theorem 4 and the claim at the end of the proof of Theorem 3 is not immediately obvious. I think it would be better if a direct argument were made in the Theorem 3 proof, or if the authors provided additional clarification regarding which part of the Theorem 4 proof was being applied.With regard to Theorem 4, it seems odd to me to state a theorem for polynomial growth graphs and then present a proof that (strictly) holds only for 2D-lattices. There isn’t a page limit on the supplementary material and the extension “Similar properties can be shown for polynomial growth graphs,…” is a little flimsy.A similar comment applies to the proof of Theorem 7; I think it is reasonable to focus on the Gaussian case, because the extension to the Poisson model seems relatively straightforward, but I can’t see a good reason to include only a sketch of the proof for the 2D-lattice when claiming a general result.


Sparse PCA with Oracle Property
In this paper, we study the estimation of the k-dimensional sparse principal subspace of covariance matrix Σ in the high-dimensional setting. We aim to recover the oracle principal subspace solution, i.e., the principal subspace estimator obtained assuming the true support is known a priori. To this end, we propose a family of estimators based on the semidefinite relaxation of sparse PCA with novel regularizations. In particular, under a weak assumption on the magnitude of the population projection matrix, one estimator within this family exactly recovers the true support with high probability, has exact rank-k, and attains a s/n statistical rate of convergence with s being the subspace sparsity level and n the sample size. Compared to existing support recovery results for sparse PCA, our approach does not hinge on the spiked covariance model or the limited correlation condition. As a complement to the first estimator that enjoys the oracle property, we prove that, another estimator within the family achieves a sharper statistical rate of convergence than the standard semidefinite relaxation of sparse PCA, even when the previous assumption on the magnitude of the projection matrix is violated. We validate the theoretical results by numerical experiments on synthetic datasets.

The abstract is an accurate summary of the paper. The paper proposes two new estimators for a sparse principal component subspace. The estimators are based on a novel regularization of a know semidefinite programming relaxation of the  rank-constrained principal subspace problem. Consistency-like properties of the estimators are shown with faster rates that previously known (nearly optimal in some regimes). Suitable optimization algorithms are described and tested on synthetic experiments, showing improved accuracy over previous methods. The new estimators also have weaker structural assumptions on the covariance matrix to guarantee convergence. The asymptotic distribution of one of the estimators is derived, making it possible to construct confidence regions.The results are fairly interesting, the presentation is clear, the results seem quite original. The significance follows from the improved theoretical rates shown as well as the improved performance in synthetic experiments.Specific comments:82: optimizaiton83: optima -> optimum141: Confusing notation: the empirical covariance matrix is S as well as \hat \Sigma. Later, S denotes the support.213: What is L?220: "If" sentence is incomplete.