Market Scoring Rules Act As Opinion Pools For Risk-Averse Agents
A market scoring rule (MSR) -a popular tool for designing algorithmic prediction markets -is an incentive-compatible mechanism for the aggregation of probabilistic beliefs from myopic risk-neutral agents. In this paper, we add to a growing body of research aimed at understanding the precise manner in which the price process induced by a MSR incorporates private information from agents who deviate from the assumption of risk-neutrality. We first establish that, for a myopic trading agent with a risk-averse utility function, a MSR satisfying mild regularity conditions elicits the agent's risk-neutral probability conditional on the latest market state rather than her true subjective probability. Hence, we show that a MSR under these conditions effectively behaves like a more traditional method of belief aggregation, namely an opinion pool, for agents' true probabilities. In particular, the logarithmic market scoring rule acts as a logarithmic pool for constant absolute risk aversion utility agents, and as a linear pool for an atypical budgetconstrained agent utility with decreasing absolute risk aversion. We also point out the interpretation of a market maker under these conditions as a Bayesian learner even when agent beliefs are static.

Summary of the paper:The paper aims to address the question of how to combine beliefs to a central judgement in the case that the beliefs held by agents have an information asymmetry in each agents perspective on the future outcomes of particular stochastic events.In particular the paper particularly focuses on the concept of Hanson 2003 known as the prediction market based on a logarithmic market scoring rule. The market scoring rule framework is sometimes also known as a market studied in the context of cost function based market makers. The motivation provided for the MSR setting involves financial markets in which market makers or dedicated specialists are provided incentives, typically in the form of rebates to take trades on both sides of the book ie. the place limit and market orders in such a way that they create liquidity in the market to facilitate buying and selling of an asset. In this context it is argued that a MSR effectively acts as a cost function-based market maker always willing to take the other side of a trade with any willing buyer or seller, and re-adjusting its quoted price after every transaction.The key results of this paper is to develop a new unified understanding of the information aggregation characteristics of a market with risk-averse agents mediated by a MSR, with no regard to how the agents' beliefs are formed. Furthermore, the authors are able to demonstrate that the MSR adjusted market is equivalent to an opinion pool.The main result from a technical perspective are provided in Theorems 2 and 3. Theorem 2 applies to the setting of myopic agents with utility functions of a particular form given by the CARA constant absolute risk aversion form that produce under certain restrictions the equivalence between the markets updated price and the result formed by a logorithmic opinion pool between the current price and the agents subjective beliefs. Note - no budget constraints were utilised - please comment on the implications of this. Theorem 3 applies to the case of decreasing absolute risk aversion for the myopic agents utility.In both cases Bayesian formulations are provided which produces a nice link with the Machine learning literature. Quality:The paper is a high quality paper with an excellent exposition of existing results and newly developed results. It attacks an existing class of problems in an interesting and non-trivial manner, whilst managing to present the results clearly and succinctly.Clarity:The paper is well written and clear. The literature review and contextual background to the work is well developed. All results are clearly defined and explained in detail.Originality:Distinct from other frameworks and existing results in behavioral finance the results developed in this paper do not presuppose any kind of generative model for agent signals, and also do not involve an equilibrium analysis. Consequently, this means that they may be used as tools to analyze the convergence characteristics of the market price in non-equilibrium situations with potentially fixed-belief or irrational agentsIn particular the work takes the approach that they consider the characteristics of the evolution of the price process instead of the properties of prices in equilibrium. In particular, examining the way in which the microstructure noise induces aggregation even when the agents are not considered in a Bayesian manner.By construction an MSR will elicit the subjective probabilities of a risk-neutral agent in an incentive compatible manner but in this context the author show that generally MSR's will elicit risk-neutral probabilities when they interact with risk-averse agents.Significance:The paper studies a well established trading environment microstructure framework in order to establish relationships between traditional belief aggregation methods such as opinion pools and Bayesian inference in two cases that are important from the perspective of the agents utility functional forms being parameterically known, ie. the widely studied CARA class and the decresing absolute risk aversion case.


Tractable Bayesian Network Structure Learning with Bounded Vertex Cover Number
Both learning and inference tasks on Bayesian networks are NP-hard in general. Bounded tree-width Bayesian networks have recently received a lot of attention as a way to circumvent this complexity issue; however, while inference on bounded tree-width networks is tractable, the learning problem remains NP-hard even for tree-width 2. In this paper, we propose bounded vertex cover number Bayesian networks as an alternative to bounded tree-width networks. In particular, we show that both inference and learning can be done in polynomial time for any fixed vertex cover number bound k, in contrast to the general and bounded tree-width cases; on the other hand, we also show that learning problem is W[1]-hard in parameter k. Furthermore, we give an alternative way to learn bounded vertex cover number Bayesian networks using integer linear programming (ILP), and show this is feasible in practice.

The paper describes a new class of Bayes nets for which inference and structure learning can be done in polynomial time.This is the class of Bayes nets with a bounded vertex cover number.So far, the only other class of Bayes nets for which inference and structure learning is tractable is the class of trees.Hence, this is an important contribution that advances our understanding of tractable probabilistic graphical models.The paper also describes two algorithms to find the best Bayes net structure for a bounded vertex cover k, however it is not clear whether practitioners would want to use those algorithms.For a fixed k, those algorithms are polynomial in the number of nodes n, but in practice it is not clear what k to use. So one would probably start with a small k and then increase k if there is some time left.Unfortunately, the algorithms scale exponentially with k, so the search would have to be restricted to small k's. It is not clear either why one would restrict the search to Bayes net with a bounded vertex cover.I understand that a bounded vertex cover implies that inference will be tractable, but this is also the case for Bayes nets with a bounded tree width. So it seems to me that practitioners would want to do the largest search possible in the space of Bayes nets that allow tractable inference. Since that space is too large for an exhaustive search, then it would make sense to focus on structures that are likely to arise in practice.It is not clear that Bayes nets with a bounded vertex cover are more likely to occur in practice than other types of tractable Bayes nets with an unbounded vertex cover.As explained in the paper, Bayes nets with a bounded vertex cover tend to have a star shape with a dense core, but I'm unaware of real problems with such a structure. To verify whether it is better to prioritize the search by increasing the bound on the vertex cover instead of the tree width, could you report two curves (one for prioritizing the search based on the vertex cover number and the other based on the tree-width) that show how the accuracy improves with time?   The paper is clear and well written.I noticed one minor typo. On the first line of page 3, the width of a tree decomposition should be *max_i* |X_i|-1.The experiments include many benchmarks, which is great.However it is difficult to tell what are the trends in those large tables of numbers.Can you bold the best result for each problem and then indicate how many problems each algorithm ended up winning?Can you also report the standard error for K-fold cross validation or some statistical test, otherwise it is not clear when some results are really better.


Secure Multi-party Differential Privacy
We study the problem of interactive function computation by multiple parties, each possessing a bit, in a differential privacy setting (i.e., there remains an uncertainty in any party's bit even when given the transcript of interactions and all the other parties' bits). Each party wants to compute a function, which could differ from party to party, and there could be a central observer interested in computing a separate function. Performance at each party is measured via the accuracy of the function to be computed. We allow for an arbitrary cost metric to measure the distortion between the true and the computed function values. Our main result is the optimality of a simple non-interactive protocol: each party randomizes its bit (sufficiently) and shares the privatized version with the other parties. This optimality result is very general: it holds for all types of functions, heterogeneous privacy conditions on the parties, all types of cost metrics, and both average and worst-case (over the inputs) measures of accuracy.

This paper looks at the multiparty computation problem and shows that if the computation must be done under differential privacy, each party should essentially use randomized response independently on its own data. The main approach is to use a linear program to model the problem. They must do a fair bit of work to characterize the optimum policy, which is randomized response.This is a "heavy" review, although I do not have too many comments for this paper. I did not have adequate time to fully verify the proofs in the supplementary material.1)In (1) it might be worth clarifying again that the input has no probability distribution so this is not average over data sets. This is important because the discussion before (8) on protocol compatibility makes it seem like the input bits are in fact random.2)One issue I have with the manuscript is that the description of the analysis in the main document seems mostly like an advertisement for how hard and challenging it is without providing enough intuition or summary to really understand what is going on. Cramming a full journal paper into the NIPS format is hard, but I would encourage the authors to think of the reader who might want a little more formal vs. descriptive exposition. That is, there is no insight into the "novel transformation" mentioned before Section 3. It would be nice to see some of that transformation described in the main document.3) Similarly how might one give more depth to the statement "[w]e need to utilize the convex geometry of the problem in order to show that interaction is not necessary"?In the end is this geometric insight intuitive or counterintuitive?4) The fact that the optimality of non-interaction holds only for single bits is, to my mind, a major drawback of this work, and is in fact rather hidden in the exposition in the beginning of the paper, where a highlighted statement "non-interactive randomized response is exactly optimal" is made that seems to be very general. This is a kind of false advertising, since the single-bit case is made to seem like a toy example that contains the insights as opposed to perhaps a pathological example where parties have extremely limited information and few options. To be clear, I think this is an interesting result, but I think one should present it plainly, warts and all.5) Finally, the main concern I have with this paper is the fit with NIPS. This is fundamentally a paper about MPC and not really about learning, machine or otherwise. While there are some intersections, this feels very much like a TCS paper and may be of more interest, more impactful, and better received in a venue such as NIPS. That being said, I think it's a solid and interesting piece of work.Smaller issues: p2: The Sweeney paper [37] did not reveal the medical record of the governor. Please read the paper and state more accurately what was actually disclosed. p5-6: The sentence going across the page is not grammatical. p6: "but it turns out" p6: "the the" --> "the" AFTER AUTHOR FEEDBACK: I thank the authors for their responses. I think my score stands as-is. I'm still not convinced about the fit, especially given the limitation to 1-bit protocols (and indications that it does not extend to multi-bit protocols). I don't think the audience at NIPS will be as appreciative.


Spatial Transformer Networks
Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.

This work addresses the question of how to improve the invariance properties of Convolutional Neural Networks. It introduces the so-called spatial transformer, a layer that performs an adaptive warping of incoming feature maps, thus generalizing the recent attention mechanisms for images. The resulting model requires no extra supervision and is trained back-to-back using backpropagation, leading to state-of-the-art results on several classification tasks. The paper is clearly written and its main contribution, the spatial transformer layer, is valuable for its novelty, simplicity and effectiveness. The related work section covers most relevant literature, except perhaps recent works that combine deformable parts models with CNNs (see for example "Deformable Part Models are Convolutional Neural Networks", "End-to-End Integration of a Convolution Network, Deformable Parts Model and Non-Maximum Suppression" both at cvpr 2015), since they also incorporate an inference over deformation or registration parameters, as in the spatial transformer case. The numerical experiments cover some relatively challenging datasets (although not large scale tasks such as Imagenet or general object recognition) which highlight the gain produced by the registration layers. Here are some detailed comments:--> relationship with a general bilinear model. The spatial transformer applies a linear transform A(x,theta(x)) to the input feature map x, which is itself indexed by a low-dimensional vector of parameters theta(x) that are adapted to the input. This is a particular instance of a general bilinear map, in which a given family of transformations (warpings or affine maps) is considered. Looking at the experiments, it seems as if the transformer is asking for more capacity, since the richer class (thin plate spline) systematically improves over simpler models of transformations. One could then attempt to make the bilinear model a bit more expressive by replacing the class of warping operators by a learnt subfamily of linear operators. Another related comment is whether it would be interesting to consider 3D warpings across feature maps as well. That could perhaps allow the network to model relationships across features with extra capacity.  --> relationship with data augmentation. In order to increase the invariance/stability of the network to large displacements or transformations out of the scope of the pooling layers, one typically resorts to data augmentation. How does the spatial transformer relate to this method? can they work together? One could argue that data augmentation uses prior information, whereas the spatial transformer does not assume classes to be invariant to known transformations. --> Another comment is whether one could use the learnt registration parameters to construct adversarial examples/data augmentation. In the model, there is a relationship between the so-called localisation parameters theta(x) and x. If say x'= T_beta(x), where T_beta is assumed to belong to the family currently implemented by the spatial transformer,then it is reasonable to expect to have theta(x') = theta(x)+beta, assuming also a group structure in the space of transformations. In any case, the localisation net should satisfy T_{theta(x)} T_beta = T_{theta( T_beta(x) ) } for all x and beta. Could this property help regularize the localisation net? --> The paper presents arguments in favor of the spatial transformers together with experiments that show its efficiency. However, the authors do not mention any scenario where the method could fail, or even hurt the performance of vanilla CNNs. When does it fail, or what are its limitations? For instance, aliasing can be potentially dangerous, as pointed out briefly, but also bilinear interpolation is known to be a bad kernel if one needs to preserve high frequency content, which might be important in, say, texture classification or complex object recognition tasks.


Private Graphon Estimation for Sparse Graphs *
We design algorithms for fitting a high-dimensional statistical model to a large, sparse network without revealing sensitive information of individual members. Given a sparse input graph G, our algorithms output a node-differentially private nonparametric block model approximation. By node-differentially private, we mean that our output hides the insertion or removal of a vertex and all its adjacent edges. If G is an instance of the network obtained from a generative nonparametric model defined in terms of a graphon W , our model guarantees consistency: as the number of vertices tends to infinity, the output of our algorithm converges to W in an appropriate version of the L 2 norm. In particular, this means we can estimate the sizes of all multi-way cuts in G. Our results hold as long as W is bounded, the average degree of G grows at least like the log of the number of vertices, and the number of blocks goes to infinity at an appropriate rate. We give explicit error bounds in terms of the parameters of the model; in several settings, our bounds improve on or match known nonprivate results.

The main theoretical contributions are: (i) in the finite graph setting, the authors provide an analysis of the L2 estimation quality of symmetric probability matrices using an ideal stochastic block model estimator (ii) the analysis is extended to a consistency analysis of the graphon parameter in a sparse graph generative model (combined with a sample-size dependent sparsity parameter), again using the L2 loss (iii) for both cases the analysis is extended to propose a mechanism for differentially private graph estimation. Each of these results is interesting and important in its own right, and advance the state of the art under what seem like reasonable conditions. Overall, the manuscript well written and conveys the main ideas, though it is clearly compressed from a full version. The "longer version" mentioned in the manuscript they refers to the supplement - which is an extended version of the submitted paper. While I like that the supplement is self contained, this is non-standard for a conference paper. If this paper is accepted, I suggest that the authors modify the appendix to the standard format. - Some standard elements e.g. a conclusion, statement of future work, and some (possibly preliminary) experimental evaluation are not included, I imagine due to limited space, or differences in writing styles/expectations across communities?- The metric on line 219 seems to me comparing a matrix A with a graphon function W using the L2 error. This does not make sense to me. I suspect it is simply a mistake in the definition. Should the definition be min_{A'} || W[A']-W ||?- It will be useful for the authors to specify explicit conditions (if any) under which the function line: 308 is normalizable as a density.Minor comments:- The definitions of L_p norm in lines 209, 210 are quite non-standardize. the authors use the p^th power of the standard definition. I suggest the authors clarify this choice in the manuscript.- Line 234: Authors should note which theorem in [7] they refer to.- Could the authors expand on an explicit algorithm for sampling from the density (line: 308) e.g. an importance sampling or Metropolis-Hastings based approach?- Line 284: "small" g' should be "capital" G.


A Generalization of Submodular Cover via the Diminishing Return Property on the Integer Lattice
We consider a generalization of the submodular cover problem based on the concept of diminishing return property on the integer lattice. We are motivated by real scenarios in machine learning that cannot be captured by (traditional) submodular set functions. We show that the generalized submodular cover problem can be applied to various problems and devise a bicriteria approximation algorithm. Our algorithm is guaranteed to output a log-factor approximate solution that satisfies the constraints with the desired accuracy. The running time of our algorithm is roughly O(n log(nr) log r), where n is the size of the ground set and r is the maximum value of a coordinate. The dependency on r is exponentially better than the naive reduction algorithms. Several experiments on real and artificial datasets demonstrate that the solution quality of our algorithm is comparable to naive algorithms, while the running time is several orders of magnitude faster.

PAPER SUMMARYThe paper proposes a generalization of submodular cover to integer lattices. Specifically, the problem is to minimum a sub-additive function c over an integer lattice, under the constraint that the value of a DR-submodular function f is at least \alpha. The paper presents an efficient greedy algorithm, which provides a multiplicative bound on the objective c, and a multiplicative violation of the constraint f. Even in the case where c is DR-submodular, and the problem can be reduced from integer lattices to set functions, the worst-case runtime of the algorithm is better than the one suggested by this reduction. COMMENTS- The generalization of submodular cover is interesting, and nicely justified through some plausible applications.- The paper is clearly written. However, section 1.1 contains several terms that are defined later. This makes it difficult to immediately follow the related work section that compares the guarantees of the proposed method with previous algorithms for special cases. Either all the terms should be specified in section 1.1, or the related work section should be moved to the end of the technical section.- When c is DR-submodular, how does the theoretical guarantee obtained by the naive reduction compare to the proposed method? The paper only mentions that the proposed method is faster. Is it as accurate?- Can multiple submodular constraints be handled using the proposed method?- The functions c and f used in the experiments are hand-tuned, and their validity for the applications is not clear. However, for the setting used, the proposed method seems to perform more accurately than some baselines, and faster than others, thereby providing a desirable trade-off.- Minor: equation (5): explain the notation "x*: optimal solution". By context, I think this means that "x* \in X*" where X* is the set of all optimal solutions. Is that right?


On-the-Job Learning with Bayesian Decision Theory
Our goal is to deploy a high-accuracy system starting with zero training examples. We consider an on-the-job setting, where as inputs arrive, we use real-time crowdsourcing to resolve uncertainty where needed and output our prediction when confident. As the model improves over time, the reliance on crowdsourcing queries decreases. We cast our setting as a stochastic game based on Bayesian decision theory, which allows us to balance latency, cost, and accuracy objectives in a principled way. Computing the optimal policy is intractable, so we develop an approximation based on Monte Carlo Tree Search. We tested our approach on three datasets-named-entity recognition, sentiment classification, and image classification. On the NER task we obtained more than an order of magnitude reduction in cost compared to full human annotation, while boosting performance relative to the expert provided labels. We also achieve a 8% F 1 improvement over having a single human label the whole set, and a 28% F 1 improvement over online learning.
"Poor is the pupil who does not surpass his master." -Leonardo da Vinci

This paper presents a hybrid approach for using both     crowdsourced labels and an incrementally (online) trained model     to address prediction problems; the core idea is to lean heavily     on the crowd as the system is ramping up, learn from the labels     thus acquired, and then use the crowd less and less often as the     model becomes more confident. This is done via a sophisticated     framing of the problem as a stochastic game based on a CRF     prediction model in which the system and the crowd are both     players. The system can issue one or more queries q for tokens x     (with true label y) which elicit responses r, where there is a     utility U(q,r) for each outcome; the system thus attempts to     pick the actions that will maximize the expected     utility. Furthermore, the queries are not issued all at once,     but at times s (with response times t); utility is maximized     with respect to a t_deadline by which an answer needs to be     computed (this thus determines how many queries are sent out, at     what rate, etc.)Computing this expected utility requires using     the simulation dynamics model P(y,r,t|x,q,s) in order to compute     the utilities as in (4). Given the utility values, the optimal     action could be chosen; however, the introduction of continuous     time makes this intractable to optimize and as such an     approximation is used based on Monte Carlo Tree Search and TD     learning (Algorithm 1). Experiments were conducted on a     simplified named entity recognition (NER) task, a sentiment     recognition task, and a face identification task, using four     methods: the majority vote of n human judges (1,3,5), online     learning (using the true labels), the "threshold" baseline (the     authors' model but without continuous time, in which m queries     are sent out at each step until the model prediction's     uncertainty is reduced below a threshold), and finally the     authors' full model (LENSE). In terms of precision/recall/F1,     the full model outperforms all but the 5-vote crowdsourced     version, though the "threshold" baseline does nearly as well     (and with lower latency for NER; Table 2). The authors also show     how the model requires fewer and fewer crowdsourced labels over     time given a sufficiently strong model to train (see Figure 3),     and that compared to online learning the accuracy is high from     the first example (Figure 4), since the system can leverage     crowd expertise more heavily while the model uncertainty is     still high.      This was a really interesting paper, and one that I expect could     generate both a lot of discussion and further work as well as     adoption in practice. There have been a variety of heuristic     approaches to learning from the crowd while training a system,     but this is the first complete and principled proposal I have     seen. The results against reasonable baselines are     impressive. As such, I feel the work is highly original and     significant, and in my mind deserves to be included in the     conference. That said, it does not fare nearly as well on the     clarity front, which unfortunately could detract from the     paper's potential impact. Many of these issues are correctible     (even for the camera-ready) if the authors are willing to put in     the time and effort and perhaps somewhat adjust the tone with     respect to the "threshold" baseline vs. the full continuous time     version.      The primary issue with clarity comes from the introduction of     continuous time, beginning partway through Section 3. The     motivation is reasonable, i.e., it might be necessary to operate     on a time (as well as a cost) budget for each classification,     but experimentally it seems to have very little benefit, while     the cost is that the formulation and discussion (as well as the     optimization procedure) become substantially more complex.In     fact, in Table 2, it appears that the "threshold" baseline     achieves significantly lower latency on the NER task while still     getting roughly equivalent performance (marginally worse); on     the face ID task it actually performs better (though still     roughly equivalent). The authors argue (l. 314) that the     baseline does better on the second task because it's a single     label prediction task and doesn't have significant interaction     between labels - however, the NER task, which has such     interaction in spades, only seems to benefit marginally from     this.For many practical tasks, the "threshold" baseline will     be good enough, and is already such a signficant gain (in terms     of being a principled hybrid learning system that outperforms     active and online learning).The paper would likely have     greater impact if the authors made this clear in the     development, i.e., the core method could be developed fully     (without the introduction of time, and as such would be easier     to understand and implement), and the addition of continuous     time could be shown as an extension.In the same vein, when     discussing the results, the authors could be more clear that the     performance of the "threshold" method is quite strong, even in     cases where there is a sequence dependence between tasks. This     is a suggestion and not strictly necessarily, as I feel the     paper is strong enough for inclusion as it is, but I do feel it     would improve the paper and the chances for the technique being     widely understood and adopted.      There are a number of smaller issues with respect to the results     and how they are reported. First, the introduction states that     the model "beats human performance" - but this is not true in     the 5-vote case for NER; strangely the 5-vote case is missing     for the Face-ID and Sentiment tasks in multiple tables and     figures (the right half of Tables 2, Table 3 and Figure 4) - it     really should be included. Likewise, in Table 2, the results for     the 5-vote case should be bolded as they represent the best     performance. The latency value for Threshold in NER should also     be bolded for the same reason (and the LENSE value for Face     ID). More importantly, the "Threshold" baseline is missing for     the sentiment experiment, and as such doesn't appear in Table 3     or Figure 4 - again, it should really be included as well.            A few minor comments with respect to the method description:            -While there is a discussion of t_deadline in l.183-187,     it is not clear what if any t_deadline was used in the     experiments, and what criteria was used to determine when the     answer was ready to report an answer - was it an entropic     threshold as with the baseline or was it reachingsome t_deadline?            -In equation (2), it seems the distribution needs to be given     the variable s as well (i.e., p(y,r,t|x,q,s).      -In equation 5, the use of q in F(q) is *very* confusing - I     assumed this referred to a query q, but in fact it just     represents a nominal distribution function - please choose some other     letter, as there are plenty to choose from.      -In the description of the Threshold baseline (l.246-252), I     assume the the expression should be (1 - p_theta(y_i|x))*0.3^m     <= (1-0.88) (note the RHS) as opposed to >= 0.88, as increasing     m will reduce uncertainty, not increase it.


Deep learning with Elastic Averaging SGD
We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

This paper studies general-purpose training algorithms for deep learning and proposes a family of algorithms called elastic averaging SGD. The idea is novel and the paper is of very high quality.The paper focuses on training large-scale deep learning models under communication constraints. This problem is difficult since there are many local optima in non-convex problems like in deep learning. The optimization problem is formulated as a global variable consensus problem such that local workers would not fall into different local optima, and then its gradient update rules are reinterpreted using the elastic forces between local and global parameters. This is why the proposed family of algorithms is named elastic averaging SGD (EASGD). There are four algorithms in the family: a synchronous version, an asynchronous version, and their momentum versions. They are demonstrated promising theoretically and experimentally.In the theoretical analysis, the authors present a vivid illustrative example in which the popular ADMM can be unstable and its stability condition is unknown, but the stability condition of EASGD is quite simple. The writing style here is very impressive and friendly. All theorems are in the supplementary material and there is only a small part of theoretical analysis in the main paper, but the organization made me feel quite smooth (though I did not carefully go through the supplementary material). I have some minor questions here: What is the most important reason for the superior stability of EASGD? Is it because the maps of ADMM are asymmetric while the maps of EASGD are symmetric? Is it possible to find another illustrative example such that on this specific example EASGD is more unstable than ADMM?In their experiments, the proposed algorithms were used to train convolutional neural networks and they outperformed state-of-the-art SGD algorithms on CIFAR and ImageNet. The proposed algorithms achieved the goal such that the larger the communication periods are, the better EASGD would be than other SGD as shown in Figure 2. Figures 3 and 4 also showed that the speedup of EASGD would be better than the baselines when there were more workers. As further experimental results, we have implemented similar gradient update rules adapted to our clusters (according to the arxiv version) and they successfully improved our baselines. The resulted models have been or will be launched online and one fifth of the world's population would benefit from these improved models. So I believe that this is a significant paper.


Recovering Communities in the General Stochastic Block Model Without Knowing the Parameters
The stochastic block model (SBM) has recently gathered significant attention due to new threshold phenomena. However, most developments rely on the knowledge of the model parameters, or at least on the number of communities. This paper introduces efficient algorithms that do not require such knowledge and yet achieve the optimal information-theoretic tradeoffs identified in Abbe-Sandon '15. In the constant degree regime, an algorithm is developed that requires only a lower-bound on the relative sizes of the communities and achieves the optimal accuracy scaling for large degrees. This lower-bound requirement is removed for the regime of diverging degrees. For the logarithmic degree regime, this is further enhanced into a fully agnostic algorithm that simultaneously learns the model parameters, achieves the optimal CH-limit for exact recovery, and runs in quasilinear time. These provide the first algorithms affording efficiency, universality and information-theoretic optimality for strong and weak consistency in the SBM.

The paper presents two algorithms for community recovery of nodes for graphs generated from stochastic block models (SBM). Two different regimes of SBM has been considered, one with constant average degree and and another with average degree of the order log(n). For the two cases, algorithms have been proposed based on shortest path finding. Main results of the paper arebounds on partial recovery of node labels with high probability for sparse (constant degree) regime and complete recovery of node labels in semi-sparse (log(n)-average degree) regime above the information-theoretic threshold of partial and complete recovery. The error bounds are given based on the proposed algorithms and shown to be optimal. The paper builds on a previous paper (Abbe and Sanders, 2015), which presented similar algorithms and analysis but with the restriction that knowledge of model parameters were needed. In this paper those restrictions are not needed anymore.The paper along with the supplementary materials is quite mathematically rigorous. Technically, the paper is of high quality. It presents strong results on almost exact recovery and exact recovery of community labels without assuming knowledge of model parameters. However, the paper lacks a bit in numerical comparison of performance with other methods of community detection.However, the main drawback of the paper is clarity. The paper presents two different algorithms, but, the algorithms are not properly outlined in the paper. The bulk of the paper including the main algorithm statements is in the supplementary material and without reading through the supplementary material, it is very difficult to parse the paper. The paper reads like a journal paper which has been truncated to meet the page constraints. The agnostic-sphere-comparison algorithm, which is the main algorithm of the paper, need to be presented a bit more clearly, like the way it has been presented in the supplementary material. Also, note that definitions of SBM is not consistent between Theorem 1 and Corollary 1 (one use Q and another Q/n) and and the way SBM has been defined states Q as probability (in line 47, page 1), which is inconsistent with the normalizations of Q made later in the paper. The paper presents an original algorithm of approximately linear time and gives a mathematically rigorous proof that the algorithm works for community recovery for general SBM. This makes the algorithms presented in the paper as one of the state-of-the-art methods for community recovery. Algorithms based on shortest paths has been previously proposed in literature (Bhattacharyya and Bickel, 2014), but in the current paper the algorithm is much more subtle and the results obtained are tighter.The paper is significant in terms of the information theoretic bounds presented in the paper as well as the algorithms introduced. The paper can be generalized to analyze more general network models, specially for growing number of communities. But, due to the technical subtlety of the paper, I feel that the paper is more suited for a journal publication than a short conference publication.


Space-Time Local Embeddings
Space-time is a profound concept in physics. This concept was shown to be useful for dimensionality reduction. We present basic definitions with interesting counter-intuitions. We give theoretical propositions to show that space-time is a more powerful representation than Euclidean space. We apply this concept to manifold learning for preserving local information. Empirical results on nonmetric datasets show that more information can be preserved in space-time.

This paper proposes a new embedding structure for the output of nonlinear dimensionality reduction (NLDR). It borrows the space-time concept from physics and shows that the new structure can encode (or represent) more information than conventional Euclidean space.The idea is pioneering and wonderful. It goes beyond all conventional embedding structures and has theoretical guarantee to accommodate more symmetry in the input space. It brings much flexibility to the NLDR problem. This can be an important middle step towards better dimensionality reduction.However, how to visualize the space-time structure seems unsolved. 1) The authors proposes to use colors to show the time dimension. But it is pretty hard to read such display. Information visualization should take advantage of the human eye's broad bandwidth pathway into the mind to allow users to digest large amounts of information at once. But Figs. 1 and 2 are rather messy. There is no immediate and clear macro pattern in these figures. Users have to examine the details piece by piece, which is time-consuming.2) the meaning of the time dimension in data science remains unclear in many aspects. For example, in Fig. 1, what is the different meanings of "very red" and "very green" (e.g. Hinton vs. Bengio)? In Fig. 2, what are the meaning of the red groups and blue groups?The KL-divergence is used as benchmarks to show the advantage of using space-time. This is not convincing because there are many other factors that can affect the results. For example, the benefits may vanish when another input kernel width or another optimization algorithm is used. In addition, the SNE and t-SNE objectives are not scale invariant to Y. That is, multiplying a same constant to all y's may change the objective. A smaller divergence may not bring a better visualization. So I don't agree Table 1 is a solid justification.The title is a bit ambiguous. Readers may think this is to find embeddings of spatial-temporal input data. How about "Local Embeddings using Space-Time"?Line 341: what do you mean "an author or a word A at a different time with the majority is close to them all"?Line 346: what do you mean "dark color"?  After feedback: The author feedback is a bit disappointing. Repeatedly arguing the advantages is not very helpful. I believe the idea is new and interesting. It deserves further exploration. But the visualization part needs improvement. From the feedback, I feel that visualizing the information in a space-time is unsolved. I don't agree with the 3D visualization approach because 1) it is restricted to only R^{2,1} case and 2) it is not provided in the paper or supplemental material. Using colors or levels to show the time dimensional is not satisfied either. The authors admit that the visualization work is still ongoing. The time dimension is claimed to show the importance (or ranking) of data points. But this remains unclear in Figure 2. The authors did not answer my question about the meanings of red words and blue words. Furthermore, it seems only the absolute values of time dimension is important. We still don't know the meaning with the signs. For example, Hinton and LeCun are green (blue in the final version), but Bengio is very red. Why? The feedback cannot explain this. The authors admit that selecting the right evaluation measure for visualization is an open problem. Therefore smaller KL-divergence values only partially demonstrate the advantages. It is not "a solid justification" as claimed. As I said, the reduced KL is not necessarily caused by the use of space-time and it may not correspond to better displays.After seeing the final version: The authors have removed most improper claims in the original version. They also improved the quality of the paper. But most of the above questions are not answered yet. They deserve further studies in future research.


Adversarial Prediction Games for Multivariate Losses
Multivariate loss functions are used to assess performance in many modern prediction tasks, including information retrieval and ranking applications. Convex approximations are typically optimized in their place to avoid NP-hard empirical risk minimization problems. We propose to approximate the training data instead of the loss function by posing multivariate prediction as an adversarial game between a loss-minimizing prediction player and a loss-maximizing evaluation player constrained to match specified properties of training data. This avoids the non-convexity of empirical risk minimization, but game sizes are exponential in the number of predicted variables. We overcome this intractability using the double oracle constraint generation method. We demonstrate the efficiency and predictive performance of our approach on tasks evaluated using the precision at k, the F-score and the discounted cumulative gain.

The approach presented in this paper - adversarial learning under multivariate performance measures - seems very interesting and original. The presentation is very clear and appropriately gradual also for a reader like myself who was new to the framework. The method is intriguing in its own right; in addition, the experimental results show that, with appropriate design choices, it can outperform structural SVM.In my opinion, the paper could be published as is. I just add some comments that should help its readability for a broader audience:1. It is true that the paper addresses a "structured prediction" problem. This is because the use of multivariate performance measures "couples" the samples even if they are i.i.d.. However, it seems to me that in the literature the term "structured prediction" is universally understood as the structuring of the output labels, i.e., feature functions which factorise over graphs, trees and the like. The paper does not address this case. In fact, the 2005 paper from Thorsten Joachims does not use "structured" or "structural" in the title. I therefore (warmly) suggest changing the title of this paper to something like "Adversarial learning under multivariate losses". The possible structuring of the feature functions can be mentioned in the body.2. On a similar rationale, it took me a bit to map the word "score" as used in the paper to the "performance measure" (although it is clearly stated at lines 062-063). It is just not very intuitive since most people associate "score" with \theta * \phi(x,y). Why not change word "score" to "perf"? Then, loss becomes 1 - perf.3. Table 1 is really a boon since it allowed me to grasp the technique. It would help to say explicitly that, here, \check{y} is the prediction and \hat{y} acts as the ground truth.4. I find the justification for eq. (7) too brief. Please add some passages to explain how the l.h.s. in the constraint becomes like this. In addition: I think \tilde{P}(x) should be removed.5. In section 3.5, before the gradient in \theta, please write the complete objective with a brief explanation.Only typo found: p.6, line 362: "demonstate"


Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks
Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.

This paper proposes using HF for learning a MDRNN with a CTC loss on top. The contributions of this work are to show that a loss that is non-convex (CTC) can still be used within Hessian Free by means of decomposing the loss using similar tricks than Gauss Newton matrices (which are applied already in HF methods for NNs). The authors show an elegant connection between their effort to make CTC suitable for HF, and an EM procedure.The authors of this paper showed an effective way to train a very complex model (MDRNN), with a non-linear cost (CTC) using second order information (HF). The main theoretical contribution is to make CTC compatible with HF by proposing a GGN that is PSD, a requirement needed for CG within HF. Their contribution goes beyond that as they show interesting links with EM, and provide empirical evidence that the method is working.I, however, have a few concerns regarding the experiments presented in this work:-As the authors discuss, TIMIT is mostly an overfitting game and not so much and underfitting / optimization game. What was the training cost achieved by both methods?-There are other baselines that may be more practical (i.e. less implementation details that may go wrong), such as Adagrad, RMSProp, or Adam. Could the authors provide such baselines? -It would be useful to add some prior art in the tables for comparison.As a final note, there are quite a lot of grammatical and minor mistakes in the text. I'd suggest a deep pass over those before publishing this work.


Deep Temporal Sigmoid Belief Networks for Sequence Modeling
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherited from the previous SBNs in the sequence, and is used to regulate its hidden bias. Scalable learning and inference algorithms are derived by introducing a recognition model that yields fast sampling from the variational posterior. This recognition model is trained jointly with the generative model, by maximizing its variational lower bound on the log-likelihood. Experimental results on bouncing balls, polyphonic music, motion capture, and text streams show that the proposed approach achieves state-of-the-art predictive performance, and has the capacity to synthesize various sequences.

I enjoyed reading the paper. It presents some interesting applications of stochastic variational inference which distinguish it from the usual paper. Methodologically, the paper defines a recurrent sigmoid belief network to model sequential data. Since maximum likelihood learning of the model is difficult, the authors propose to use a variational setup, where the approximating distribution q is defined as another directed model. A lower bound on the log likelihood of data is maximized by alternatively maximizing the joint distribution p(v,h) under samples h from q, and maximizing parameters of q in equation 11 (which I believe is equivalent to minimizing the KL between posterior of p and q, although its never stated that way). The optimization is done with Stochastic Gradient Descent on Monte Carlo samples (as was popularized in Stochastic Variational Inference by Hoffman, Blei et. al and other papers predating NVIL) since exact analytical treatment is not possible.The results are presented on four different datasets with interesting results. With regards to the results, I was surprised that the TSBN performed quite poorly on bouncing balls dataset with an order of 1 (pred err 13.23), but so well with order of 4 (pred err = 3.07). To me this implies that the model has not done a very good job of learning long range structure, through the hidden units. No results are presented for the deep TSBM with order 1. And yet these are compared to the temporal restricted boltzmann machine, which I believe used order 1. Either ways, it would seem that RTRBM actually performs better on bouncing balls than this model if order 1 is used. For Motion Capture results it would have been good to also report results from conditional RBM models (although, I believe the MSE results are worse than those in table 2). The conditional RBMs models videos seem a lot less jittery than the ones attached in supplementary methods for this paper. The presentation of section 5.4 [Prediction] is a little incomplete. It isn't stated what the modeling task is clearly, or what the data is that is modeled at each time step. Presumably its the word distribution at each time step, and this information is available in the supplied reference [31]. However it would be good to have at least a one line description here, space permitting. Similarly for the Dynamic Topic Modeling subsection.   Quality: Paper is well presented, and methods are demonstrated on datasets with quite different characteristics, showing that the models is versatile enough. Clarity: Quite clear presentation of ideas. I would suggest one thing. Wrt equation 7, authors should state that this procedure optimizes the exact bound if q is exactly the posterior of p. Also, it should be stated that optimizing equation 11 is equivalent to minimizing the KL of posterior of p from q. For clarity, a one line note with equation 11 can be added to state that sum_h q(h) d/dtheta log q(h) is 0 and drops out, in equation 11.Originality: The novelty of the paper is in defining the temporal SBM and pairing it with the recognition network. The other novelty is in the application to the different datasets. The methods themselves aren't entirely new, but are an application of neural stochastic variational inference to this model. However, the paper is novel enough, in my opinion, since it shows this method can learn. Significance: The paper extends the range of successful applications to which stochastic variational inference ideas have been applied, and is hence useful.Nevertheless, it would have been useful to compare with other variational idea that do sequence learning (Learning Stochastic Recurrent Networks by Bayer and Oserndorfer).


Subsampled Power Iteration: a Unified Algorithm for Block Models and Planted CSP's
We present an algorithm for recovering planted solutions in two well-known models, the stochastic block model and planted constraint satisfaction problems (CSP), via a common generalization in terms of random bipartite graphs. Our algorithm matches up to a constant factor the best-known bounds for the number of edges (or constraints) needed for perfect recovery and its running time is linear in the number of edges used. The time complexity is significantly better than both spectral and SDP-based approaches. The main contribution of the algorithm is in the case of unequal sizes in the bipartition that arises in our reduction from the planted CSP. Here our algorithm succeeds at a significantly lower density than the spectral approaches, surpassing a barrier based on the spectral norm of a random matrix. Other significant features of the algorithm and analysis include (i) the critical use of power iteration with subsampling, which might be of independent interest; its analysis requires keeping track of multiple norms of an evolving solution (ii) the algorithm can be implemented statistically, i.e., with very limited access to the input distribution (iii) the algorithm is extremely simple to implement and runs in linear time, and thus is practical even for very large instances.

This paper gives a new, simple algorithm for block models (especially when the graph is bipartite and the two sides are highly imbalanced), and show that the problem of planted CSP can be reduced to this case. As a result this paper gives simple "spectral"-like algorithms for planted CSPs for all k, where previously for odd k the only known results require SDP or more complicated spectral analysis.This is a very interesting paper. The new algorithm is clean and efficient. The reviewer feels this gives significant better understanding to planted k-CSPs when k is odd. The main algorithmic technique is based on simple power iteration, but the subsampling idea is important (as otherwise it becomes the spectral approach which we know will not work in certain cases). The proof relies on careful induction on the mean and variance of the entries.Minor points: Definition 3: In order for this to be useful there must also be a distance to r-wise independent (or in other words lowerbound in the fourier coefficient), would be nice to discuss how that appears in Theorem 2 below.Section 5 explained why spectral algorithms cannot work, it would be nice to also explain why subsampling helps, as it still looks very much like power method that computes the top singular vectors. The induction proof does not give much intuition.Overall I think this paper is very nice and should be accepted.


Fast Classification Rates for High-dimensional Gaussian Generative Models
We consider the problem of binary classification when the covariates conditioned on the each of the response values follow multivariate Gaussian distributions. We focus on the setting where the covariance matrices for the two conditional distributions are the same. The corresponding generative model classifier, derived via the Bayes rule, also called Linear Discriminant Analysis, has been shown to behave poorly in high-dimensional settings. We present a novel analysis of the classification error of any linear discriminant approach given conditional Gaussian models. This allows us to compare the generative model classifier, other recently proposed discriminative approaches that directly learn the discriminant function, and then finally logistic regression which is another classical discriminative model classifier. As we show, under a natural sparsity assumption, and letting s denote the sparsity of the Bayes classifier, p the number of covariates, and n the number of samples, the simple ( 1 -regularized) logistic regression classifier achieves the fast misclassification error rates of O s log p n , which is much better than the other approaches, which are either inconsistent under high-dimensional settings, or achieve a slower rate of O s log p n .

I have read the paper ``fast classification rates for high-dimensional conditional Gaussian models".The paper studiesthe problem of binary classificationusing a Gaussian model and provides some theoretical results on the convergence of the classification error rates (compared to the Bayes classifier). The paper presents some nice theoretical results and is interesting to some extent. I am generally positive about the paper but I have the following concerns. First, it is about the practical relevance.One of the very important point about classification is that we need a tuning free trained classifier; when a classifierhas tuning parameters that need to be tuned, we usually use data splittingto decide such tuning parameters. Therefore, it is more relevant to either studythe error rates of a tuning free classifier, or that of a classifier with tuning parametersselected by data splitting.  This is important for that, while the convergence rate of the "ideal tuning parameters"may be fast, the convergence rate of the classifier that is actually used could be very slow, simply because of the convergence of the data-driven tuning parameters to the ideal tuning parameters are slow.  In terms of this, the paper needs to (a) compare more carefully with recent tuning-free classifiers, (b) address the effect of selecting the tuning parameters by data-splitting. Second of all, the paper has put all the focus on the convergence rate,does notcontain any insightful comparison of the computing speed.The penalization methodsthe authors present are all computationally slow when $p$ is large. It is desirable tohave a detailed comparison about the computational complexity or speed. Third of all, given this is a well-studied topics, it is desirable to comparewith different methods using at least 1-2 real data sets (say, microarray data sets). The methods that need to compare include: those discussed in the paper [3,7,13,15,17] if the method is easy to implement as well as support vector machine (SVM) and random Forest. Fourth, the paper comments that assuming $\Sigma^{-1} (\mu_1 - \mu_0)$ is sparse is weaker than assuming both $\Sigma^{-1}$ and $(\mu_1 - \mu_0)$ are sparse. Mathematically, this is true. However, the more relevant point is that, whether we have an important application example(say, in genomics or genetics) where $\Sigma^{-1} (\mu_1 - \mu_0)$ is sparse but either $\Sigma^{-1}$ is non-sparse or$(\mu_1 - \mu_0)$ is non-sparse. If we don't have such application examples in areas many people care, then arguing how two assumptions are different from each other is largely due to mathematicalcuriosity, not due to scientific interest. Last, the presentation can be improved. The material can be streamlined so that the main contribution can be more highlighted.


On Elicitation Complexity
Elicitation is the study of statistics or properties which are computable via empirical risk minimization. While several recent papers have approached the general question of which properties are elicitable, we suggest that this is the wrong question-all properties are elicitable by first eliciting the entire distribution or data set, and thus the important question is how elicitable. Specifically, what is the minimum number of regression parameters needed to compute the property? Building on previous work, we introduce a new notion of elicitation complexity and lay the foundations for a calculus of elicitation. We establish several general results and techniques for proving upper and lower bounds on elicitation complexity. These results provide tight bounds for eliciting the Bayes risk of any loss, a large class of properties which includes spectral risk measures and several new properties of interest.

=======POST REBUTTAL AND DISCUSSION COMMENTS: I seem to be in the minority in thinking the paper is too technical for the NIPS crowd. I am raising my score accordingly. =======This paper considers the problem of property elicitation. A property of a probability distribution is simply a (scalar) function of this distribution, and a property G is said to be elicitable if there exists a loss function such that the minimizer of the expected loss for a distribution P coincides with G(P). The authors observe that the ability to elicit a function of a distribution is something of a weak notion, since one can always elicit the full distribution and then return the property value. Instead, the important question is to ask *how many* parameters would need to be elicited in order to produce the property value for all distributions. The paper provides some nice results that help to understand the notion of "elicitation complexity", and the paper gives some applications of these results that connect with recent work. The paper also considers a new notion of "conditional elicitation" and provides some limited exploration of this idea.On the positive side, the paper does seem to provide nice novel results that help to understand elicitation, making nice connections with the estimation of the Bayes risk. The framework the authors put forth is coherent and natural, and builds upon a great deal of previous but recent work on this topic. The paper really does bring out a coherent way to reason about elicitability. There seem to be cool tricks in here too; in particular, I noticed that Theorem 1 characterizes a class of scoring rules that require an increasing function, a tool I've not seen previously.On the negative side, and why I think this paper is really going to be a stretch for NIPS, is that: (a) the work is highly technical, (b) the paper would be readable by only a small subset of the NIPS community, (c) the technical parts are quite dense with very little in the way of examples or helpful discussion, (d) the 8-page format is too small to lay out the framework appropriately, with motivating examples etc. Overall i think the authors really aren't doing justice to this work if they want to sell this to the broader ML community. This paper, after expanding and improving the technical content, would be much more appropriate for COLT or a statistics journal. I'm probably a 5.5 on this paper, but I could be swayed based on the response and the feelings of the other reviewers.More detailed comments: -- The authors use terms like "how much effort is required to compute it via ERM" but I think this confuses the notion of effort. The "complexity" discussed in this paper is more about description length and not about computational complexity or sample complexity. Those are also interesting questions but not really addressed. -- The beginning of Section 2 is way too dense. The reader not familiar with this line of work is going to find this impossible to wade through. -- Why does 'elic' have a subscript of \mathcal{I} but 'iden' does not? -- The reason that elic(\Gamma) >= iden(\Gamma) should be done much more explicitly, I still don't understand why it's obviously true. -- The authors appear to combine properties via {}-notation, but {} are usually used for sets and so this is confusing. -- I found Theorem 2 quite hard to parse. -- footnote 7 has Example 3.1.4 mentioned twice. -- On the one hand, section 4 seems very dense and quite rushed. On the other hand, there's space at the bottom of page 8 which suggests there was room to add more explanation.


A Gaussian Process Model of Quasar Spectral Energy Distributions
We propose a method for combining two sources of astronomical data, spectroscopy and photometry, that carry information about sources of light (e.g., stars, galaxies, and quasars) at extremely different spectral resolutions. Our model treats the spectral energy distribution (SED) of the radiation from a source as a latent variable that jointly explains both photometric and spectroscopic observations. We place a flexible, nonparametric prior over the SED of a light source that admits a physically interpretable decomposition, and allows us to tractably perform inference. We use our model to predict the distribution of the redshift of a quasar from five-band (low spectral resolution) photometric data, the so called "photoz" problem. Our method shows that tools from machine learning and Bayesian statistics allow us to leverage multiple resolutions of information to make accurate predictions with well-characterized uncertainties.

The paper presents a latent variable model for modeling spectral energy distribution of quasars, given stereoscopic and photometric observations. The joint modeling of stereoscopic and photometric measurements allows the model to make inferences about stereoscopic properties of quasars leveraging the more broadly available photometric data. Clarity: The paper for the most part is well written and easy to follow. I have some minor complaints about the exposition, see detailed comments below. Originality: Moderate. The authors develop a well motivated, non trivial latent variable model for capturing the salient properties of distributions of noisy quasar measurements. The use of parallel tempering in the inference procedure is interesting as well.  Significance: Although an application paper, the work is interesting and could spur further interest among the NIPS community, in tackling these important computational problems in astrophysics. Detailed comments:Model: 1) It would be useful to be explicit about the domains and the priors of the various random variables.      a) Is red shift variable z_n, constrained to take any real value in the domain [0, 7]. Is the prior on z_n, a uniform distribution in this interval?      b) What is the domain of the apparent brightness variable m_n, the positive real line? More importantly what prior was placed on m_n? Line 217 alludes to an empirically derived prior, but no further details are provided.    c) T he Dirichlet distribution is a standard prior on a collection of non negative variables that sum to one.What is the more complicated mixture of log-Gaussian priors over w buying us here? A stronger motivation for this non standard, more complicated choice of prior will be helpful. Inference: I found some of the details of the inference procedure confusing.1) Basis estimation: It appears that the point estimate of the bases is being arrived at by finding a MAP estimate of an auxiliary model, one that ignores the photometric data. Why is the true model posterior which incorporates the photometric likelihood term p (y | w, z, m, B) in equation 10, not optimized instead? The MAP estimates of the two models are obviously different. When inferring w, z, and m, why does it make sense to condition on the MAP basis estimates of the auxiliary model? 2) It is interesting that the authors resort to parallel tempering. It is certainly plausible that a naive MCMC sampler will mix poorly. However, it would be good to clearly demonstrate the need for parallel tempering and show how much worse a single slice sampler chain (with T_c = 1) really is, in terms of red shift predictions. Perhaps an extra row could be added to Figure 5 and an additional line plot in Figure 6 to depict prediction results with and without parallel tempering.Experiments: I like the two data splits with different train and test brightness and redshifts, mimicking real world scenarios.The results could be better explained.1) In figure (5) does the y-axis correspond to shifts inferred by the model? The rightmost scatter plot seems weird. 2) I found figure (6) very confusing.It is never referenced in the text and it isn't clear what is being depicted in it. Are we looking at predicted SED (conditioning on photometric observations) for two held out quasars? Is the noisy line plot in light blue the observed SED for these quasars? What is the mysterious PCA-based model displayed in black, which seems to be fitting the data much better. Is it better because it is "cheating" and has been fit to all available SED's including the two SEDs displayed here?


Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation
We propose a novel deep neural network architecture for semi-supervised semantic segmentation using heterogeneous annotations. Contrary to existing approaches posing semantic segmentation as a single task of region-based classification, our algorithm decouples classification and segmentation, and learns a separate network for each task. In this architecture, labels associated with an image are identified by classification network, and binary segmentation is subsequently performed for each identified label in segmentation network. The decoupled architecture enables us to learn classification and segmentation networks separately based on the training data with image-level and pixel-wise class labels, respectively. It facilitates to reduce search space for segmentation effectively by exploiting class-specific activation maps obtained from bridging layers. Our algorithm shows outstanding performance compared to other semi-supervised approaches with much less training images with strong annotations in PASCAL VOC dataset.

This paper describes a method of semantic segmentation, that can be applied particularly in cases where there are many image-level class labels but few segmented images.Classes are first identified using a classification network, then individually segmented using a second segmentation network.The second network is fed with the upper convolutional feature map of a the classification network, as well as the back-propagated activation of the individual class, which acts as a sort of seed.These are passed through a set of fc layers, which are reshaped and passed to a generating ``deconvolutional'' network that produces the segmentation for the class.Evaluations are performed using Pascal VOC.Overall, this pieces together some good components to create a nice segmentation system.However, some of the design choices could be further explored, and it would be nice to compare on more datasets than only Pascal VOC 2012.Questions follow:* I believe the same segmentation network is applied to all classes (i.e. weights are shared between classes), from l.219.But how much does this fact contribute to its performance when using few strongly annotated images, compared to the combinatorial data augmentation?* How many channels does the bridging layer output g_i^l have?Fig 2 suggests just 1.* While Fig 2 shows the bridging layer output is similar for each class, I wouldn't necessarily conclude that the "search space may not be as huge as our prejudice".Another possibility is that the class pattern is used in the first layer of the deconvolutional network to essentially perform the same unpooling (with switches) operation that forms the backpropagated map f^l_cls, so that the supplying the class provides more information than the spatial component (which is largely redundant).* Related, what happens if only f^l_cls is used (no f_spat), or if it is substituted with the class using a 1-hot encoding?What is the contribution of using this particular representation?* The text indicates the segmentation network is applied to each "identified label" -- how are these labels identified precisely from the classification net outputs?* A few additional references for semantic segmentation with convnets, which might also be included: - Farabet et al. "Learning Hierarchical Features for Scene Labeling" TPAMI 2013- Eigen & Fergus "Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture" arxiv 2014- Wang et al. "Deep Joint Task Learning for Generic Object Extraction" NIPS 2014


Parallel Recursive Best-First AND/OR Search for Exact MAP Inference in Graphical Models
The paper presents and evaluates the power of parallel search for exact MAP inference in graphical models. We introduce a new parallel shared-memory recursive best-first AND/OR search algorithm, called SPRBFAOO, that explores the search space in a best-first manner while operating with restricted memory. Our experiments show that SPRBFAOO is often superior to the current state-of-the-art sequential AND/OR search approaches, leading to considerable speed-ups (up to 7-fold with 12 threads), especially on hard problem instances.
obtained without using sophisticated schemes, as done in previous work [8]. An extensive empirical evaluation shows that our new parallel recursive best-first AND/OR search scheme improves considerably over current state-of-the-art sequential AND/OR search approaches, in many cases leading to considerable speed-ups (up to 7-fold using 12 threads) especially on hard problem instances.

The paper is straightforward in a good way. RBFAOO is briefly described and then an appropriately detailed description of SPRBFAOO is given. It is clear that some thought (and no doubt experimentation) has gone into the presented design. As it should be, there is quite a long section on empirical results which help the reader understand the important issues.It would be interesting to see whether a "programming by optimisation (PBO)" approach would yield good results - perhaps eg fitting \zeta to features of a particular problem instance.In [2] RBFAOO is only compared against AOBB and AOBF, so although it is superior to both those algorithms it is not clear to me how it compares to alternative graphical model MAP algorithms such as those which participated in the UAI'14 inference challenge (which had a MAP competition). Note that the proteus solver, a portfolio solver, did well (beating daopt in the 20 and 60 min competitions). It included CPLEX which automatically parallelises if there are the processors available, so that's another example of parallel search for MAP. I acknowledge that the parallelisation strategy here is more sophisticated and problem-specfic than sending the problem to a general-purpose solver that happens to be able to run in parallel.I suspect this paper would be of interest to a fairly narrow pool of NIPS people since it has a narrow focus. It is no surprise that, if done competently, using more computational power can speed up (a particular approach to )MAP solving in MRFs.typos, etc:reduces the frequency of occurring the scenarios -> reduces the frequency of scenarios


Probabilistic Line Searches for Stochastic Optimization
In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.

One of the challenges of stochastic optimization with mini-batch gradient descent on, e.g., deep neural networks, is the selection of an appropriatelearning rate.In deterministic optimization, this choice is often avoided by using line search.This paper proposes a way to model the noise of stochastic objectives when performing line search and use Bayesian optimization to intelligently perform that search.It shows that careful choices for the covariance function can lead to efficient searches and that standard Wolfe conditions can be framed probabilistically.The paper provides recommendations for various meta-parameters in the algorithm.Empirical evaluation is performed on two-layer neural networks and show that the optimization is robust to initial learning rate choices.I love the idea of this paper.I've hoped someone would write this paper for a while, as it has become clear that Bayesian optimization should have something to say about traditional line search.I commend the authors for their thoughtful consideration of a variety of practical issues, from covariance choice, to the subtleties of weak vs strong Wolfe conditions, as well as their willingness to make practical recommendations about meta-parameters.My major complaint about the paper is that it does not follow through and examine whether this additional work provides a win over the increasingly-large literature on adapting learning rates.I really want this to be the "right" way to solve these kinds of problems, but I did not come away from the paper feeling like I should definitely use this.That is, here are some questions that I wanted to see empirically answered, but were not: - How sensitive is the method to variations in c_1, c_2, c_w, and \tau?- How does this method compare to algorithms that have scale adaptation such as  those in in [8], [9], [10], and [14]?- The choice of mini-batch size (10) is unusually small, and yet this has a  large impact on the noise.What is the impact of this choice?Other Comments and Questions: - The Wolfe conditions are a particular heuristic for balancing improvement in  the line search against starting a new one, given limited information about  the function.When exploring new probabilistic ideas, is it really necessary  to take this heuristic that seriously as a gold standard?- When the estimators y_t and y'_t are being computed, or when they are  computed for different values of t in the line search, is the same subset of  the data re-used?This was not clear to me, but the independence assumption  stated in Eq. (3) makes me think that they are different subsets.- What does it mean for an objective to be "irregular"?- I spent some time drawing samples from the covariance function in Eq. (4) and  did not find myself thinking these were interesting hypotheses for  objectives.They tended to be straight lines with small perturbations. Also,  I found that \tau had a large impact; one expects that for large values of  \tau, the min functions would essentially just return \tau.Thus \tau seems  to impact the effective scale of the problem.I wanted a more thoughtful  discussion of this issue.- Given that the paper is motivated by issues with line search for stochastic  objectives, a well-referenced and evidence-supported discussion of the  "brittleness" discussed in the introduction would've been helpful.Moreover,  it seems two things are ultimately conflated in the paper: 1) using Bayesian  optimization for line search in stochastic _or_ deterministic problem, and 2)  Bayesian optimization as a way to address the brittleness of line search in  stochastic optimization.- I don't really buy the need for analytic identification of the minima of the  mean function because the whole point of the Wolfe conditions is to avoid  the need to find exact minima; various heuristics for finding the minima  using, e.g., a grid, would be fine given the other costs involved.Also, it  is somewhat worrying to make potentially mismatched modeling decisions simply  to guarantee that it is easy to find the minimum between the evaluations.- I like the story told by Figure 2.- The explanation of c_w=0.3 was potentially interesting and should go into the  supplementary material.I suspect the authors have a careful argument to  make here that instead comes across as hand-wavy.Also, please evaluate the  empirical sensitivity to this choice.Presentational Issues:- P1 L32: "... if their gradient is corrupted ..." to "... if their gradients  are corrupted ..."- P1 L34: What is an "exchangeable" loss function?- P1 L36-40: The notation where j=1..m and also "indices j are i.i.d. draws"  doesn't really make sense and could perhaps be made more clear.- P1 L40: "... the error ... is unbiased ..." should probably be some more like  "... the estimator \hat{\mathcal{L}} of \mathcal{L} is unbiased ..."- P2 Figure 1: I found this figure somewhat confusing.I suggest drawing the  "true" function using a light color to make the decisions clearer.It would  be helpful if the value of c_1 was mentioned in the caption.I am not sure  that I understand the representation of the Wolfe conditions; is the idea  that if the tangent lines go through the orange regions then that point is  not satisfactory?- P3 L146-153: The notation y_i was previously used to denote the data in  Section 1 and it is here being reused to indicate noisy function evaluations.- P3 L133-153: The notation f is introduced as the objective, but the objective  was previously introduced as \mathcal{L} with the \hat{\mathcal{L}} being the  noisy estimate now represented as y.I would like there to be greater  precision in the notation describing these functions.- P3 Figure 2: The top figure is very busy for how small it is.I suggest  removing the sample paths, i.e., the dashed lines.- P4 L169-192: The somewhat unusual notation for partial derivatives would  benefit from a one-sentence explanation.- P5 L231: I am assuming that \eta is the minimum of the mean over actual  previous evaluations and not candidates, but a sentence that explains this  would be helpful.- P5 L258: The probability of Wolfe conditions being satisfied is a _truncated_  bivariate normal probability, I think.- P5 Figure 3: Very little explanation of this figure in the caption and it  doesn't seem to be referred to in the text at all.- P8 Figure 4: This figure is not well explained.What do the colors mean  relative to the top and the bottom?Are the colors on the bottom meant to  refer to the different vertical columns, i.e., initial learning rates of the  top?


Tractable Learning for Complex Probability Queries
Tractable learning aims to learn probabilistic models where inference is guaranteed to be efficient. However, the particular class of queries that is tractable depends on the model and underlying representation. Usually this class is MPE or conditional probabilities Pr(x|y) for joint assignments x, y. We propose a tractable learner that guarantees efficient inference for a broader class of queries. It simultaneously learns a Markov network and its tractable circuit representation, in order to guarantee and measure tractability. Our approach differs from earlier work by using Sentential Decision Diagrams (SDD) as the tractable language instead of Arithmetic Circuits (AC). SDDs have desirable properties, which more general representations such as ACs lack, that enable basic primitives for Boolean circuit compilation. This allows us to support a broader class of complex probability queries, including counting, threshold, and parity, in polytime.

Summary of paper: The paper deals with learning SDDs (from previous work) that can be used to infer complex queries. Inference for these type of queries could not be done efficiently in previous models like the ACMNs. The paper provides details about the learning algorithm (which is similar to the ACMN learning algorithm based on greedy selection), different types of complex queries (symmetric and asymmetric that can be converted to symmetric), and experimental evaluation on real data to answer how well SDDs do and how fast they are. Complexity bounds on the inference complexity are also provided. Quality: The paper is of high quality and is well written. Did not find any errors in text or any equations. Clarity: Along with the supplemental text (description of SDDs and WMC), the paper is quite clear to follow. Suggestion but not required would be to provide similar summary text in the supplemental material for ACMN especially since you have the figure in 1(c).Originality: To the best of my knowledge, this is a novel method to learn SDDs from queries and trying to infer more difficult queries than was previously done using tractable models. Significance: Given that the paper provides another step in the direction of use of tractable models to solve even more difficult queries, there could be other ideas that could build on what is provided here.  Questions, clarifications and suggestions to improve paper:1) Are sum-product networks any better than ACMNs to solve these complex queries? If so, how would they compare against use of SDDs ?2) You use the work conjoin at multiple places, it might be help define the operator.3) In the supplementary text you mention additional unary parameters w_A and w_B ? Is there any reason they are not included in the tree 1b and 1c ? Are they not needed to be included in the SDD ?4) In Algorithm 1, you pass \alpha as a parameter. You should pass \alpha as a parameter to score(M, T) since score depends on it.5) What step of the algorithm are you doing dynamic SDD minimization ? Is it done in M.add(f) step ?6) Is it possible that the minimization or compressing can reduce the size of the SDD more in step t+2 by selecting features f1 and f2 combined compared to when you would have selected f3 and f4 sequentially in a greedy way ? i.e. does selecting features in a greedy fashion cause generating a SDD that might not be optimally minimized ?7) It would be good to have example queries where ACMN does similar to SDDs as well as asymmetric queries that are not tractable by SDDs for completeness.8) It might be better to use func() instead of # in Table 1.9) Fix opening double quotes on page 7. ... vote "yes" on a bill ...10) In the positive, negative movie review question, is there any reason you don't use all characters instead of restricting to first 750 characters ? does it affect performance and how ?11) While answering the second question posed and using complex queries on the voting and movie dataset, is there any reason you do not look at the accuracy as well as the time especially for queries that do not time out. It seemed like the Mutex queries and Complex queries you had are quite different and you didn't test for the accuracy in terms of log-likelihood on the complex queries.12) Complete the references with details (i.e. for example put in page numbers for references 3, 6, 8, 12 etc)


A Tractable Approximation to Optimal Point Process Filtering: Application to Neural Encoding
The process of dynamic state estimation (filtering) based on point process observations is in general intractable. Numerical sampling techniques are often practically useful, but lead to limited conceptual insight about optimal encoding/decoding strategies, which are of significant relevance to Computational Neuroscience. We develop an analytically tractable Bayesian approximation to optimal filtering based on point process observations, which allows us to introduce distributional assumptions about sensory cell properties, that greatly facilitate the analysis of optimal encoding in situations deviating from common assumptions of uniform coding. The analytic framework leads to insights which are difficult to obtain from numerical algorithms, and is consistent with experiments about the distribution of tuning curve centers. Interestingly, we find that the information gained from the absence of spikes may be crucial to performance.

This paper describes a tractable method for performing dynamic state estimation from point process observations. The authors derive closed-form expressions for the mean and variance of a Gaussin approximation to the posterior (using assumed density filtering), characterize the optimal placement of tuning curves, and emphasizes the importance of information carried by "silences" to the posterior.The paper is well-written, original, highly novel, and provides a very beautiful extension of previous work on optimal tuning curves for static problems. It is likely to have a substantial impact on the field and should certainly be accepted.Comments:1. The abstract and intro seem slightly misleading in implying that the solution is fully general, when the details presented here seem to require Gaussian tuning curves.(Perhaps I missed it, but do the authors have a solution for dealing with non-Gaussian tuning curves, or a posterior density that is not Gaussian?Even tuning curves given by a Gaussian plus a constant DC baseline rate would seem to create a big problem because one would no longer obtain a closed-form update to the posterior from the spike-related likelihood terms).I still think the work is extremely nice, but a little bit more of a disclaimer about what cases are considered and/or limits on the form of the code might be helpful here. (Alternatively, if they have an idea about how to handle non-zero baseline spike rate, it would helpful to hear it.)2. I'm a little bit confused about one aspec of the Gaussian approximation to the non-spiking terms in the log-likelihood.If we go a long time without seeing any spikes, the true posterior becomes bimodal due to the -f(\theta) term -- intuitively, the probability that the state is far away on one tail or the other becomes increasingly high, the longer we go without spikes.What happens to the approximation in this case - does it simply tend towards uniform?I assume this is an extreme case that the authors regard as relatively unlikely for a reasonable sized population with reasonable marginal spike rate, but would be nice to say a word or two about when the approximation will break down.3.Two refs the authors might consider citing:- Huys, Zemel, Natarajan, & Dayan, NC 2007 - relatively early paper considering the problem of decoding a dynamic posterior distribution from spike trains.- Grabska-Barwinska & Pillow, NIPS 2014 - considered optimal tuning curve problem for uniform case but for different error metrics & constraints. Minor:Fig 2 caption:"dashed green line" - doesn't appear to be green.


Sum-of-Squares Lower Bounds for Sparse PCA
This paper establishes a statistical versus computational trade-off for solving a basic high-dimensional machine learning problem via a basic convex relaxation method. Specifically, we consider the Sparse Principal Component Analysis (Sparse PCA) problem, and the family of Sum-of-Squares (SoS, aka Lasserre/Parillo) convex relaxations. It was well known that in large dimension p, a planted k-sparse unit vector can be in principle detected using only n ≈ k log p (Gaussian or Bernoulli) samples, but all efficient (polynomial time) algorithms known require n ≈ k 2 samples. It was also known that this quadratic gap cannot be improved by the the most basic semi-definite (SDP, aka spectral) relaxation, equivalent to a degree-2 SoS algorithms. Here we prove that also degree-4 SoS algorithms cannot improve this quadratic gap. This average-case lower bound adds to the small collection of hardness results in machine learning for this powerful family of convex relaxation algorithms. Moreover, our design of moments (or "pseudo-expectations") for this lower bound is quite different than previous lower bounds. Establishing lower bounds for higher degree SoS algorithms for remains a challenging problem.

Paper 997 - Sum-of-Squares Lower Bounds for Sparse PCA ======================================================The authors explore the statistical-computational trade-off in the Sparse PCAproblem within the Sum-Of-Squares convex relaxation hierarchy.Their worknicely complements many recent papers in the literature on this subject and ismost directly related to the paper by Krauthgamer, Nadler, and Vilenchik (2015). The results of KNV (2015) can be interpreted as a negative result for the SoSdegree-2 relaxation.This paper provides analogous results for the degree-4 SoS relaxation: showing that the optimal value of the the program is large with high probability, and hence prone to false rejection, and that its output isweakly correlated with the true.This paper is closer to the COLT community,but the results are very good and the paper is well-written.It should beaccepted into NIPS.I only have a few typographical corrections:page 3, line 140: "even to" -> "even" page 3, line 145: "An very" -> "A very" page 5, line 234: "once can" -> "one can" page 6, line 299: "above as ." -> "above." page 6, line 310: "Rigollet's" -> "Berthet and Rigollet's"