Learning values across many orders of magnitude
Most learning algorithms are not invariant to the scale of the signal that is being approximated. We propose to adaptively normalize the targets used in the learning updates. This is important in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.
normalization can be used to disentangle the natural magnitude of each component from its relative importance in the loss function. This is particularly useful when the components have different units, such as when we predict signals from sensors with different modalities. Finally, adaptive normalization can help deal with non-stationary. For instance, in reinforcement learning the policy of behavior can change repeatedly during learning, thereby changing the distribution and magnitude of the values.

The authors propose a method for adaptively normalizing the output distribution of the regression targets a neural network is trained to produce. The idea is to be able to elegantly deal with output targets of different scales, especially in the situation where the normalization of the targets cannot be computed a priori, as is generally the case when learning action value functions in reinforcement learning. The authors show on a toy task that their method is able to deal with low frequency targets that are much larger than the average targets. They show on the Atari 2600 reinforcement learning task that their system performs better on average than systems that use a clipping heuristic. Their model also seems to learn qualitatively different behavior because it's able to distinguish between rewards of different magnitudes.
I think the problem that the authors are trying to solve is a very important one that shows up in many gradient-based learning situations. The ideas are straightforward but (to my knowledge) new and apparently effective. For this reason I can see them becoming widely used.  The paper is well-written and at most places clear. The related work section seems to contain enough relevant references, but it would be nice if some of the most related works would be discussed in a bit more detail. The work by Wang et al. (2016), for example, seems related to me in the sense that it seems to make the DQN in that paper less sensitive to the mean of the values as well.  I like the toy example. While it seems to me that a linear model should be able to solve this task (one solution would be to set the input weights equal to powers of two), the gap between the presented data and the infrequent high target makes this impossible to learn from the data and representative of a problem that may even occur when the output targets are normalized but still follow a fat-tailed distribution.   I found it a bit difficult to judge the Atari 2600 results because they represent such a high dimensional comparison of different types of scores. The movies in the supplementary materials (especially Centipede) look convincing and the method sometimes performs much better than the baseline, but it also performs much worse on some games. On average, the method performs better and I agree with the authors that being able to learn most of the games without clipping is already a step forward. It would be interesting to see how well the method would do with at least a bit of hyperparameter tuning. Especially the discount factor. The authors also point out that the success of models that use clipping might be somewhat specific to this task. For this reason, it would have been interesting to see a comparison on another reinforcement learning task as well.  Some small things:  Line 55: The word 'most' is very imprecise for statements of this kind. Does this mean that one of the natural gradient methods is actually invariant to rescaling the targets?  Line 66: I think a more correct interval for the step size would be (0, inf) instead of [0, 1]. You clearly don't want to include 0 and I don't see why a step size couldn't be greater than 1.  Line 88: The actual 'objective' mentioned in this line is never discussed. The rescaling updates seem to be more about satisfying certain constraints like the ones presented in Equation 3 and the bound on the normalized targets.  Algorithm 2: The update of W seems to contain a typo where 'g' should probably be 'h'.


Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making
We propose Confusions over Time (CoT), a novel generative framework which facilitates a multi-granular analysis of the decision making process. The CoT not only models the confusions or error properties of individual decision makers and their evolution over time, but also allows us to obtain diagnostic insights into the collective decision making process in an interpretable manner. To this end, the CoT models the confusions of the decision makers and their evolution over time via time-dependent confusion matrices. Interpretable insights are obtained by grouping similar decision makers (and items being judged) into clusters and representing each such cluster with an appropriate prototype and identifying the most important features characterizing the cluster via a subspace feature indicator vector. Experimentation with real world data on bail decisions, asthma treatments, and insurance policy approval decisions demonstrates that CoT can accurately model and explain the confusions of decision makers and their evolution over time.

The authors propose a generative model for time-stamped K-categorical ratings specific to evaluator—item pairs.  The authors cover relevant background work in the crowdsourcing and human annotation literature in section 2.  In section 3, the authors describe the proposed CoT model which is based on clustering evaluators and items and associating a series of KxK confusion matrices with each (evaluator-cluster, item-cluster) pair.   Each series contains T confusion matrices, one for each discrete time step in the data, and a simple Markov chain is imposed over the series so that the matrices evolve smoothly through time.  In section 4, the authors describe a series of experiments where they compare the proposed model to ablations of the model and three existing baselines on prediction and estimation tasks using both real and synthetic data.
The authors motivate the proposed model with the setting in which items have “true” but unobserved labels/ratings and the observed labels/ratings given by evaluators are potentially incorrect.  This differs from the very common problem in recommendation systems or collaborative filtering where evaluators provide their subjective ratings but there is not assumed to be any “true” rating (e.g., users of Netflix giving 1-5 star ratings to movies). This seems like a common but underexplored setting that is worthy of further study within machine learning.  The authors are also right to highlight interpretability as a desired aspect of any machine learning solution that may yield post-hoc insights into common human biases and thus suggest corrective measures.  This paper does a good job of motivating the proposed model and situating it within the crowdsourcing and human annotation literature.  However the paper does not provide any coverage of two highly related areas of machine learning research.  The proposed generative model is a stochastic block model (SBM) [1, 2].  There is a vast literature on SBMs that includes many dynamic variants.  I think this paper must at least mention this literature to show that the proposed model is novel.  Furthermore, the setting of evaluator—item categorical ratings is very familiar to machine learning within the areas of collaborative filtering, recommendation systems, and matrix factorization yet the paper does not mention any of these areas.  Much of the writing is clear but the description of the model is not easy to read owing to some mistakes in the mathematical definitions and overly burdensome notation.   The binomial distribution defined on line 128 and throughout should be the Bernoulli distribution.  Similarly the multinomial distribution defined on line 138 and throughout should be the categorical distribution.  The uniform distribution has support on the real numbers and thus the uniform distribution defined on line 124 and throughout should be a categorical distribution.  Is V_{m,e} in equation 1 ever defined?  The description of the experiments is vague.  It is unclear where and when the true labels are used during training.  On line 211, it’s stated that the true labels are only used for evaluation but on line 228 it’s said that 5% are used during training.  It’s also not said how the Gibbs sampler is used to generate predictions.  Is the last sample taken as a point estimate of the latent parameters? Or are multiple samples being used?  It’s stated that the Gibbs sampler can be parallelized but not shown how.  Also, how exactly is BIC used to set the number of clusters? With cross-validation?  The experiments section in general needs to be more precise in order for the results to be understood and interpreted.   [1] Nowicki, K. and Snijders, T. A. B. Estimation and prediction for stochastic blockstructures. Journal of the American Statistical Association, 96(455):1077–1087, 2001.  [2] Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T., and Ueda, N. Learning systems of concepts with an infinite relational model. In Proceedings of the Twenty-First National Conference on Artificial Intelligence, 2006.


Can Peripheral Representations Improve Clutter Metrics on Complex Scenes?
Previous studies have proposed image-based clutter measures that correlate with human search times and/or eye movements. However, most models do not take into account the fact that the effects of clutter interact with the foveated nature of the human visual system: visual clutter further from the fovea has an increasing detrimental influence on perception. Here, we introduce a new foveated clutter model to predict the detrimental effects in target search utilizing a forced fixation search task. We use Feature Congestion (Rosenholtz et al.) as our non foveated clutter model, and we stack a peripheral architecture on top of Feature Congestion for our foveated model. We introduce the Peripheral Integration Feature Congestion (PIFC) coefficient, as a fundamental ingredient of our model that modulates clutter as a non-linear gain contingent on eccentricity. We show that Foveated Feature Congestion (FFC) clutter scores (r(44) = −0.82 ± 0.04, p < 0.0001) correlate better with target detection (hit rate) than regular Feature Congestion (r(44) = −0.19 ± 0.13, p = 0.0774) in forced fixation search; and we extend foveation to other clutter models showing stronger correlations in all cases. Thus, our model allows us to enrich clutter perception research by computing fixation specific clutter maps. Code for building peripheral representations is available 1 .

The paper proposes a new clutter model that integrates peripheral information on top of an existing clutter model (Feature Congestion). Their model was developed for predicting target detection rate, with targets situated at different eccentricities of a fixation point.  The paper also proposed an experimental design for collecting the behavioral data in a forced fixation search experiment. The results showed that the foveated version of the feature congestion model was able to better explain the target detection rates at different eccentricities than the original feature congestion model.
This work suffers from several issues in my opinion: problem statement, novelty and technical contribution, and experiments.  Problem statement: The author states that most clutter models produce a global metric to measure clutter perception, but the fovea effect have not been taken into account. Then they argue that the fovea effect should be a part of the clutter model, where the problem is framed as a target hit-rate modeling study. I wonder, why is this an interesting problem? As the current clutter models (other than feature congestion) have achieved high correlations with human clutter rankings, why would it even be important to add in the peripheral structure into an existing model, where such existing model is a good margin behind the current state-of-the-art (feature congestion). Also, the problem is centered around the “forced fixation search” experiment, which means the subject is fixating at a specified location while the target is somewhere in the image. This leads to the foveated clutter model to take in a fixation-location input parameter, in order to generate the peripheral structure as part of the pipeline. However, this is disturbing because I do not see under any circumstance would a viewer being forced to fixate at some part of an UI, control panel, advertisement, or any visual display, because in a real world scenario people perform free viewing on the displays. Therefore, in terms of the application of clutter perception models, the forced-fixation points should at least be chosen meaningfully, such as with a fixation-prediction algorithm, or obtain human fixations from an initial free-viewing experiment, then use those as the forced-fixation locations for the actual search experiment.  Novelty and technical contribution: This work has very limited technical scope, as it is addressing a very specific scope of clutter perception modeling: forced fixation search problem. The work combines an existing peripheral structure work onto an existing clutter model (feature congestion) in a very limited scope of clutter perception. The proposed forced-fixation search experiment is also very specific that does not offer sufficient general contribution to the NIPS community. In addition, the feature congestion model is a good margin behind the current state-of-the-art in predicting global human clutter perception, it would be more desirable for the authors to come up with an approach to base their method on the state-of-the-art clutter model, such as the proto-object model.   Experiments: As mentioned previously, if the authors want to build on top of an existing work, it would be more desirable to be adopting the state-of-the-art. For the experiments to be more convincing, the authors should try to propose a more general framework that is able to be combined with most of the clutter model, so that the experiments can include the foveated feature congestion, foveated proto-object model, foveated scale invariance model, etc. With the current experiments of comparing the proposed model against the original feature congestion model, the results are underwhelming because the authors are comparing both methods to the forced-fixation search data, where the original feature congestion model was never designed for that specific task anyway, so of course it would not do well as compared to the proposed method that is designed specifically to solve this issue. A better claim should be showing the foveated model can do better than other clutter models in terms of the general clutter ranking task, not at a task that is specifically designed for the foveated model. Therefore, the comparison part is weak and should include more models in a more general scope.


Deep Neural Networks with Inexact Matching for Person Re-Identification
Person Re-Identification is the task of matching images of a person across multiple camera views. Almost all prior approaches address this challenge by attempting to learn the possible transformations that relate the different views of a person from a training corpora. Then, they utilize these transformation patterns for matching a query image to those in a gallery image bank at test time. This necessitates learning good feature representations of the images and having a robust feature matching technique. Deep learning approaches, such as Convolutional Neural Networks (CNN), simultaneously do both and have shown great promise recently.
In this work, we propose two CNN-based architectures for Person Re-Identification. In the first, given a pair of images, we extract feature maps from these images via multiple stages of convolution and pooling. A novel inexact matching technique then matches pixels in the first representation with those of the second. Furthermore, we search across a wider region in the second representation for matching. Our novel matching technique allows us to tackle the challenges posed by large viewpoint variations, illumination changes or partial occlusions. Our approach shows a promising performance and requires only about half the parameters as a current state-of-the-art technique. Nonetheless, it also suffers from false matches at times. In order to mitigate this issue, we propose a fused architecture that combines our inexact matching pipeline with a state-of-the-art exact matching technique. We observe substantial gains with the fused model over the current state-of-the-art on multiple challenging datasets of varying sizes, with gains of up to about 21%.
In this work, we propose two CNN-based architectures for Person Re-Identification. In the first, given a pair of images, we extract feature maps from these images via multiple stages of convolution and pooling. A novel inexact matching technique then matches pixels in the first representation with those of the second. Furthermore, we search across a wider region in the second representation for matching. Our novel matching technique allows us to tackle the challenges posed by large viewpoint variations, illumination changes or partial occlusions. Our approach shows a promising performance and requires only about half the parameters as a current state-of-the-art technique. Nonetheless, it also suffers from false matches at times. In order to mitigate this issue, we propose a fused architecture that combines our inexact matching pipeline with a state-of-the-art exact matching technique. We observe substantial gains with the fused model over the current state-of-the-art on multiple challenging datasets of varying sizes, with gains of up to about 21%.

This paper proposes two CNN models for person re-identification. The novel aspects of the first model are normalized cross correlation (Normalized X-Corr) layer and wider search area for patch matching. The second model is the combination of the Ahmed’s model [1] and the first model. The experimental results show the substantial performance gain over the state-of-the-art deep learning based methods on three benchmark datasets.
I think the proposed method seems to be somewhat novel and the performance is impressive. However, the evaluation and the writing are clearly rushed and incomplete. Detailed comments are bellows:   Pros ) The performance of the proposed method is impressive. Even with only Normalized X-Corr model, the proposed method achieves substantially higher identification rates on CUHK01 and CUHK03 datasets than state-of-the arts.   The Normalized X-Corr layer is interesting and seems to reasonable if the performance is really gained by this layer (see Cons. comment).   This paper shows that combination of the different deep architectures improves the performance. This is simple idea, but as far as I know, this is not explored except a work [A]. I consider that the work [A] is published after the submission of this paper.   [A] F.Wang, W.Zuo, L.Lin, D.Zhang, L.Zhang, Joint Learning of Single-image and Cross-image Representations for Person Re-identification, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1288-1296, 2016.  Cons ) The main drawback of this paper is the incomplete baseline comparison. The proposed method has two differences from the Ahmed et al [1]. These are; (1) Normalized X-Corr layer and (2) wider area search. However, this paper does not report the baseline results except the reported results by Ahmed et al. [1]. Therefore, contributions of each part ((1) and (2)) are not understand. For example, the high performance of the proposed method might be comes from the wider area search and the Normalized X-Corr layer might be not contributed to the performance, and vice versa. The authors could compare the performance with the Cross Input Neighborhood layer of Ahmed’s model with wider search area, and Normalized X-Corr layer with smaller area search of Ahmed’s model, for example.   One advantage claimed in this paper “the parameters of the Normalized X-Corr model is about half number compared to the Ahmed’s model (eg. line170-171)” is incorrect. In Fig.3, the size of Normalized X-Corr layer is 12 x 37 x 1500 while the Cross Input Neighborhood layers of Ahmed’s model is 12 x 37 x 1250. Clearly, the parameters of Ahmed’s model is smaller. This paper doesn’t explain how the number of parameters comes from and why it is smaller than that of Ahmed’s model. In my understanding, the depth of Cross Input Neighborhood is comes from 5x5 (search area) x 2 (asymmetric distance) x 25 (feature maps) = 1250. While the depth of Normalized X-Corr is comes from 12x5 (search area) x 1 (symmetric distance) x 25 (feature maps) = 1500. Due to the large search area, the parameters of the proposed model is larger than that of Ahmed’s model.   The VIPeR dataset is the most commonly used dataset for person re-identification. However, the experimental results on this dataset is not reported. It would be better to report the results on VIPeR dataset.   Writing quality of the paper is low:   - Line 21. and 59: What is jumped and which datasets?  - Line 29: “1.1 Problem Definition” can be removed.  - Line 51, 158: What do you mean “images are regular” ? - Line 154: It would be better to explain how did Ahmed et al. set search space.  - Line 183 “they are fused in a 1000 node fully connected layer”: Is it two separate 500 node fully connected layers (Fig. 3) ?  - Line 250: How much did you get speed up by multi-GPU training?  - Line 255, “Normalized X-Corr model give a 3% gain”: It should be “1.63% gain”. (64.73% vs 62.1%).  - Line 275, “a gain of upto 16% over the existing state-of-the art”: Please specify which setting on the CUHK01.  - Some capitalization should be collected or unified. eg.,   “Person Re-Identification” → “person re-identification”   “Dataset” → “dataset”   “Underground Re-identification (GRID)” →“underGround Re-IDenditifcaiton (GRID)”  Supplementary material: - How did you obtain Eq.(1) ? More explanations are need.  - Please use \eqnarray (if you use latex) for Eq.(1-2) and Eq.(5-7) . - Eq.(3) should be removed.  - I do not understand your intention to show the Fig.7-8.


Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions
A single color image can contain many cues informative towards different aspects of local geometric structure. We approach the problem of monocular depth estimation by using a neural network to produce a mid-level representation that summarizes these cues. This network is trained to characterize local scene geometry by predicting, at every image location, depth derivatives of different orders, orientations and scales. However, instead of a single estimate for each derivative, the network outputs probability distributions that allow it to express confidence about some coefficients, and ambiguity about others. Scene depth is then estimated by harmonizing this overcomplete set of network predictions, using a globalization procedure that finds a single consistent depth map that best matches all the local derivative distributions. We demonstrate the efficacy of this approach through evaluation on the NYU v2 depth data set.

This paper proposed a novel monocular depth estimation algorithm. Instead of directly fitting the depth map, the method predicts a series of filter-bank response coefficients of the depth map. In the prediction stage the proposed method predict the filter response distribution and recover the depth through a global energy minimization step.
In general the proposed method is technically sound. The general idea also makes sense in better capturing high-level statistics. The method seems novel. However, I have some concerns on how to prove it matters and why the proposed method is chosen.    The general idea is somewhat similar with some recent work which incorporates loss function over the high-order statistics on many low-level vision tasks such as flow, monocular depth recovery and image generation (e.g. "DisparityNet" Mayer et al. 2016, "Image Generation with Perceptual Loss" Ridgeway et a. 2015, "DeePSiM loss" Dosovitskiy et al. 2015, "Depth Estimation" Eigen et al. 2015). All the methods use loss functions that capture high-level statistics, or so-called perceptual loss through handcrafted (or learnable) filter map responses.   Motivated by this I think a baseline is required to convince the readers: instead of fitting filter-bank response then followed by an energy minimization stage to recover the depth, whether we can fit the depth using the following loss function: the coefficient similarity between the filter responses from GT depth and predicted depth. This scheme does not require the time-consuming alternating inference in prediction, but in the meantime captures the high-order statistics through the handcrafted filters. I wonder what is the performance and why the proposed is favored by the author.  The author claims the importance of fitting the coefficient of filter-bank responses, but did not to show such importance through experiments. Is this visually more appealing or better preserve 3D geometry than directly fitting depth map? Or does modeling the uncertainty brings diverse prediction? From section 4.2 I can hardly understand why it matters. From the comparison study, it seems even much less important than pretraining a VGG model on ImageNet or incorporating semantic labeling loss.   Moreover, there lacks of some foundation why fit high-order depth map statistics directly, rather than the 3D geometry space, and why such a series of handcrafted gaussian filters are used. In fact, it is totally doable that the author could incorporate the inference stage of eq. 6 into the network and conducted learning the filters in an end-to-end manner. In this way the objective of reconstructing depth could also directly utilized. In fact, I also have a doubt that the indirect regression might hurts the performance in terms of RMSE since the model is not optimized to minimize such a loss.   Overall I think the current stage of the paper is on the borderline and I am expecting the author's rebuttal to address my concerns.


Computing and maximizing influence in linear threshold and triggering models
We establish upper and lower bounds for the influence of a set of nodes in certain types of contagion models. We derive two sets of bounds, the first designed for linear threshold models, and the second more broadly applicable to a general class of triggering models, which subsumes the popular independent cascade models, as well. We quantify the gap between our upper and lower bounds in the case of the linear threshold model and illustrate the gains of our upper bounds for independent cascade models in relation to existing results. Importantly, our lower bounds are monotonic and submodular, implying that a greedy algorithm for influence maximization is guaranteed to produce a maximizer within a 1 − 1 e -factor of the truth. Although the problem of exact influence computation is NP-hard in general, our bounds may be evaluated efficiently. This leads to an attractive, highly scalable algorithm for influence maximization with rigorous theoretical guarantees.
A series of recent papers [12,21,13] establish computable upper bounds on the influence when information propagates in a stochastic manner according to an independent cascade model. In such a model, the infection spreads in rounds, and each newly infected node may infect any of its neighbors in the succeeding round. Central to the bounds is a matrix known as the hazard matrix, which encodes the transmission probabilities across edges in the graph. A recent paper by Lee et al. [11] leverages "sensitive" edges in the network to obtain tighter bounds via a conditioning argument. Such bounds could be maximized to obtain a surrogate for the influence-maximizing set in the network; however, the tightness of the proposed bounds is yet unknown. The independent cascade model may be viewed as a special case of a more general triggering model, in which the infection status of each node in the network is determined by a random subset of neighbors [9]. The class of triggering models also includes another popular stochastic infection model known as the linear threshold model, and bounds for the influence function in linear threshold models have been explored in an independent line of work [5,23,4]. Naturally, one might wonder whether influence bounds might be derived for stochastic infection models in the broader class of triggering models, unifying and extending the aforementioned results. We answer this question affirmatively by establishing upper and lower bounds for the influence in general triggering models. Our derived bounds are attractive for two reasons: First, we are able to quantify the gap between our upper and lower bounds in the case of linear threshold models, expressed in terms of properties of the graph topology and edge probabilities governing the likelihood of infection. Second, maximizing a lower bound on the influence is guaranteed to yield a lower bound on the true maximum influence in the graph. Furthermore, as shown via the theory of submodular functions, the lower bounds in our paper may be maximized efficiently up to a constantfactor approximation via a greedy algorithm, leading to a highly-scalable algorithm with provable guarantees. To the best of our knowledge, the only previously established bounds for influence maximization are those mentioned above for the special cases of independent cascade and linear threshold models, and no theoretical or computational guarantees were known.
The remainder of our paper is organized as follows: In Section 2, we fix notation to be used in the paper and describe the aforementioned infection models in greater detail. In Section 3, we establish upper and lower bounds for the linear threshold model, which we extend to triggering models in Section 4. Section 5 addresses the question of maximizing the lower bounds established in Sections 3 and 4, and discusses theoretical guarantees achievable using greedy algorithms and convex relaxations of the otherwise intractable influence maximization problem. We report the results of simulations in Section 6, and conclude the paper with a selection of open research questions in Section 7.

The paper develops new upper and lower bounds for computing the influence of a set in the Linear Threshold cascade model, and in the the General Triggering model, which includes the Independent Cascade model as a special case. These seem to be the first such bounds for the Linear Threshold model, and the bounds for the Independent Cascade model are tighter in some cases than previous results. The analysis is based on fairly elementary "path weight" arguments. These are tighter in the Linear Threshold model, where at most one live path exists from a source to any given node, than in the General Triggering model. The paper analyzes the conditions under which the upper and lower bounds are within a constant factor of each other, and provides examples for which their bounds are tighter than previous ones (ref [16]).   The authors further show that the lower bounds are submodular, and so they can be optimized nearly optimally using the greedy algorithm as a principled surrogate for the true influence. Experiments on synthetic graphs demonstrate that the bounds are valid, and that optimizing the bounds as surrogates performs very well for the original influence maximization task.
The paper is very well written and presents a clear and straightforward analysis of bounds for the influence function in the linear threshold and triggering models.   The technical quality is high. The techniques are straightforward (elementary probability arguments) and provide bounds that appear to be useful in practice. The analysis is careful and thorough. For example, Section 3.3 includes both an example of when the bounds are bad, as well as an analysis of conditions under which the bounds are tight. The experiments demonstrate the the bounds are useful in practice for influence maximization.  The potential impact is reasonable. There continues to be interest in influence maximization and diffusion processes in the CS literature, and I can envision follow-on work or application of these methods. (Note to authors: in general I feel this line of work has become regrettably detached from real applications, so the extent to which you can connect back to real applications would strengthen future work.)  I found two areas that could use further discussion to strengthen the paper:  1) In the experiments on influence maximization, the main benefit of the proposed method is running time. It would be appropriate to compare running times of different methods to quantify the accuracy vs. running-time tradeoff of maximizing the lower bounds. (It appears there is little accuracy lost, so simply reporting running times may tell the story you need to tell).  2) Some further discussion of the comparison to the results of Lemmonier et al. would strengthen the paper. For example, can you quantify empirically the conjecture in Line 217 (condition for when your bound is tighter)? This conjecture suggests that your bound may perform best in the "sub-critical" regime (small cascades relative to graph size). I note that your experiments generally seem to have small cascade sizes (influence < 30 in graphs with 900 nodes). Do your bounds also perform well in with large cascades?   Minor notes:   Definition 1. The wording is somewhat misleading. It says that "f is submodular if either (i) or (ii)" but does not say that (i) and (ii) are equivalent conditions. Consider adding a statement to this effect for clarity.  In Section 5.2 you discuss approximately optimizing the lower bounds, but do not actually state that this problem is NP-hard and cannot be solved optimally. The results in the appendix seem to suggest this. If it is true, I suggest you state it in the main text of the paper.


Optimizing Affinity-Based Binary Hashing Using Auxiliary Coordinates
In supervised binary hashing, one wants to learn a function that maps a highdimensional feature vector to a vector of binary codes, for application to fast image retrieval. This typically results in a difficult optimization problem, nonconvex and nonsmooth, because of the discrete variables involved. Much work has simply relaxed the problem during training, solving a continuous optimization, and truncating the codes a posteriori. This gives reasonable results but is quite suboptimal. Recent work has tried to optimize the objective directly over the binary codes and achieved better results, but the hash function was still learned a posteriori, which remains suboptimal. We propose a general framework for learning hash functions using affinity-based loss functions that uses auxiliary coordinates. This closes the loop and optimizes jointly over the hash functions and the binary codes so that they gradually match each other. The resulting algorithm can be seen as an iterated version of the procedure of optimizing first over the codes and then learning the hash function. Compared to this, our optimization is guaranteed to obtain better hash functions while being not much slower, as demonstrated experimentally in various supervised datasets. In addition, our framework facilitates the design of optimization algorithms for arbitrary types of loss and hash functions.
Information retrieval arises in several applications, most obviously web search. For example, in image retrieval, a user is interested in finding similar images to a query image. Computationally, this essentially involves defining a high-dimensional feature space where each relevant image is represented by a vector, and then finding the closest points (nearest neighbors) to the vector for the query image, according to a suitable distance. For example, one can use features such as SIFT or GIST [23]  and the Euclidean distance for this purpose. Finding nearest neighbors in a dataset of N images (where N can be millions), each a vector of dimension D (typically in the hundreds) is slow, since exact algorithms run essentially in time O(N D) and space O(N D) (to store the image dataset). In practice, this is approximated, and a successful way to do this is binary hashing [12]. Here, given a high-dimensional vector x ∈ R D , the hash function h maps it to a b-bit vector z = h(x) ∈ {−1, +1} b , and the nearest neighbor search is then done in the binary space. This now costs O(N b) time and space, which is orders of magnitude faster because typically b < D and, crucially, (1) operations with binary vectors (such as computing Hamming distances) are very fast because of hardware support, and (2) the entire dataset can fit in (fast) memory rather than slow memory or disk.
The disadvantage is that the results are inexact, since the neighbors in the binary space will not be identical to the neighbors in the original space. However, the approximation error can be controlled by using sufficiently many bits and by learning a good hash function. This has been the topic of much work in recent years. The general approach consists of defining a supervised objective that has a small value for good hash functions and minimizing it. Ideally, such an objective function should be minimal when the neighbors of any given image are the same in both original and binary spaces. Practically, in information retrieval, this is often evaluated using precision and recall. However, this 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
ideal objective cannot be easily optimized over hash functions, and one uses approximate objectives instead. Many such objectives have been proposed in the literature. We focus here on affinity-based loss functions, which directly try to preserve the original similarities in the binary space. Specifically, we consider objective functions of the form min L(h) = N n,m=1 L(h(x n ), h(x m ); y nm )
where X = (x 1 , . . . , x N ) is the high-dimensional dataset of feature vectors, min h means minimizing over the parameters of the hash function h (e.g. over the weights of a linear SVM), and L(·) is a loss function that compares the codes for two images (often through their Hamming distance h(x n ) − h(x m ) ) with the ground-truth value y nm that measures the affinity in the original space between the two images x n and x m (distance, similarity or other measure of neighborhood; [12]). The sum is often restricted to a subset of image pairs (n, m) (for example, within the k nearest neighbors of each other in the original space), to keep the runtime low. Examples of these objective functions (described below) include models developed for dimension reduction, be they spectral such as Laplacian Eigenmaps [2] and Locally Linear Embedding [24], or nonlinear such as the Elastic Embedding [4] or t-SNE [26]; as well as objective functions designed specifically for binary hashing, such as Supervised Hashing with Kernels (KSH) [19], Binary Reconstructive Embeddings (BRE) [14] or sequential Projection Learning Hashing (SPLH) [29].
If the hash function h was a continuous function of its input x and its parameters, one could simply apply the chain rule to compute derivatives over the parameters of h of the objective function (1) and then apply a nonlinear optimization method such as gradient descent. This would be guaranteed to converge to an optimum under mild conditions (for example, Wolfe conditions on the line search), which would be global if the objective is convex and local otherwise [21]. Hence, optimally learning the function h would be in principle doable (up to local optima), although it would still be slow because the objective can be quite nonlinear and involve many terms.
In binary hashing, the optimization is much more difficult, because in addition to the previous issues, the hash function must output binary values, hence the problem is not just generally nonconvex, but also nonsmooth. In view of this, much work has sidestepped the issue and settled on a simple but suboptimal solution. First, one defines the objective function (1) directly on the b-dimensional codes of each image (rather than on the hash function parameters) and optimizes it assuming continuous codes (in R b ). Then, one binarizes the codes for each image. Finally, one learns a hash function given the codes. Optimizing the affinity-based loss function (1) can be done using spectral methods or nonlinear optimization as described above. Binarizing the codes has been done in different ways, from simply rounding them to {−1, +1} using zero as threshold [18,19,30,33], to optimally finding a threshold [18], to rotating the continuous codes so that thresholding introduces less error [11,32]. Finally, learning the hash function for each of the b output bits can be considered as a binary classification problem, where the resulting classifiers collectively give the desired hash function, and can be solved using various machine learning techniques. Several works (e.g. [16,17,33]) have used this approach, which does produce reasonable hash functions (in terms of retrieval measures such as precision/recall).
In order to do better, one needs to take into account during the optimization (rather than after the optimization) the fact that the codes are constrained to be binary. This implies attempting directly the discrete optimization of the affinity-based loss function over binary codes. This is a daunting task, since this is usually an NP-complete problem with N b binary variables altogether, and practical applications could make this number as large as millions or beyond. Recent works have applied alternating optimization (with various refinements) to this, where one optimizes over a usually small subset of binary variables given fixed values for the remaining ones [16,17], and this did result in very competitive precision/recall compared with the state-of-the-art. This is still slow and future work will likely improve it, but as of now it provides an option to learn better binary codes.
Of the three-step suboptimal approach mentioned (learn continuous codes, binarize them, learn hash function), these works manage to join the first two steps and hence learn binary codes [16,17]. Then, one learns the hash function given these binary codes. Can we do better? Indeed, in this paper we show that all elements of the problem (binary codes and hash function) can be incorporated in a single algorithm that optimizes jointly over them. Hence, by initializing it from binary codes from the previous approach, this algorithm is guaranteed to achieve a lower error and learn better hash functions. Our framework can be seen as an iterated version of the two-step approach: learn binary codes given the current hash function, learn hash functions given codes, iterate (note the emphasis).
The key to achieve this in a principled way is to use a recently proposed method of auxiliary coordinates (MAC) for optimizing "nested" systems, i.e., consisting of the composition of two or more functions or processing stages. MAC introduces new variables and constraints that cause decoupling between the stages, resulting in the mentioned alternation between learning the hash function and learning the binary codes. Section 1 reviews affinity-based loss functions, section 2 describes our MAC-based proposed framework, section 3 evaluates it in several supervised datasets, using linear and nonlinear hash functions, and section 4 discusses implications of this work.
Related work Although one can construct hash functions without training data [1,15], we focus on methods that learn the hash function given a training set, since they perform better, and our emphasis is in optimization. The learning can be unsupervised [5,11], which attempts to preserve distances in the original space, or supervised, which in addition attempts to preserve label similarity. Many objective functions have been proposed to achieve this and we focus on affinity-based ones. These create an affinity matrix for a subset of training points based on their distances (unsupervised) or labels (supervised) and combine it with a loss function [14,16,17,19,22]. Some methods optimize this directly over the hash function. For example, Binary Reconstructive Embeddings [14] use alternating optimization over the weights of the hash functions. Supervised Hashing with Kernels [19] learns hash functions sequentially by considering the difference between the inner product of the codes and the corresponding element of the affinity matrix. Although many approaches exist, a common theme is to apply a greedy approach where one first finds codes using an affinity-based loss function, and then fits the hash functions to them (usually by training a classifier). The codes can be found by relaxing the problem and binarizing its solution [18,30,33], or by approximately solving for the binary codes using some form of alternating optimization (possibly combined with GraphCut), as in two-step hashing [10,16,17], or by using relaxation in other ways [19,22].

The authors propose a solution to the data-sensitive locality preserving hashing problem for retrieval applications, in the standard framework.  y_{nm} is the ground truth similarity between items n, m, and x_n, x_m, their intrinsic representations.  The authors need to fit a hash function h such that h(x_n) and h_(x_m) are similar or dissimilar as dictated by y_{nm}.  This means the loss is written as L(h(x_n), h(x_m), y_{nm}), i.e., h nests inside L.  The authors approach this optimization by declaring auxiliary (binary) variables z_n which are regularized toward h(x_n) gradually using a suitable term in the objective.
The authors propose a solution to the data-sensitive locality preserving hashing problem for retrieval applications, in the standard framework.  y_{nm} is the ground truth similarity between items n, m, and x_n, x_m, their intrinsic representations.  The authors need to fit a hash function h such that h(x_n) and h_(x_m) are similar or dissimilar as dictated by y_{nm}.  This means the loss is written as L(h(x_n), h(x_m), y_{nm}), i.e., h nests inside L.  The authors approach this optimization by declaring auxiliary (binary) variables z_n which are regularized toward h(x_n) gradually using a suitable term in the objective.  Procedurally, it is an unsurprising alternating optimization: hold the hash functions fixed and optimize the z_n's, then hold the z_n's fixed and refit the hash functions.  But the authors claim (I haven't checked the proofs) some interesting theoretical results about the trajectory of solutions with the regularization.  And they also present some competent experiments (with nice post-mortem analysis), although the experimental outlay is somewhat modest.  A million images is fine (I would be ok with even 100,000 images), but what would be more interesting is how the system performs under more heterogenous clustering/similarity situations, as in real image search hosted by commercial search companies. (L270) Why did you choose 1:5 positive:negative neighbors? How did you choose other hyper/nuisance parameters like kernel SVM regularization etc.?  The work is clearly competent, but I am not in a position to judge with high confidence how impactful it is in the community of recent work.  The paper was a delight to read; lamentably few papers are written so clearly nowadays.  It was easy to understand by non-experts, was not overly loud with claims --- a pleasure!


Local Minimax Complexity of Stochastic Convex Optimization
We extend the traditional worst-case, minimax analysis of stochastic convex optimization by introducing a localized form of minimax complexity for individual functions. Our main result gives function-specific lower and upper bounds on the number of stochastic subgradient evaluations needed to optimize either the function or its "hardest local alternative" to a given numerical precision. The bounds are expressed in terms of a localized and computational analogue of the modulus of continuity that is central to statistical minimax analysis. We show how the computational modulus of continuity can be explicitly calculated in concrete cases, and relates to the curvature of the function at the optimum. We also prove a superefficiency result that demonstrates it is a meaningful benchmark, acting as a computational analogue of the Fisher information in statistical estimation. The nature and practical implications of the results are demonstrated in simulations.

The paper introduces the notion of local minimax complexity for stochastic convex optimization (SCO). The setting they consider is very similar to the classical notion of (first order) SCO: one wishes to minimize a unknown function f from some family F, and one may repeatedly query an oracle with a point x, and the oracle returns a subgradient of f at x with Gaussian noise (they have an additional assumption that the oracle always returns the same subgradient, but this is not a big deal).   The traditional goal in this setting is then to approximately minimize f using as few oracle calls as possible. The traditional minimax risk is max over f in F of min over algorithms A of the expected error that A makes on f. That is, the algorithm needs to work to minimize all f in the class. In contrast, in this paper, they fix a function f and they define the local minimax complexity of optimizing f the max over g in F of min over algorithms A of the expected error A makes on either g or f. That is, the algorithm needs only differentiate between f and g (and these are known to the algorithm). In some sense, this captures a notion of "instance-by-instance" optimality for SCO.  The paper then shows that when F is the set of Lipschitz convex functions, the local minimax risk is exactly captured by a notion of modulus of continuity that they define. However, the algorithm which achieves the upper bound here is not actually useful for optimization. They also show a "superefficiency" result: that any algorithm which outperforms the local minimax risk of a function at some function must perform substantially worse on some nearby function. Finally, they give an algorithm which achieves local minimax risk for SCO in 1D.
This is a very interesting paper, and a novel attempt to capture a notion of per-instance optimal rates for SCO. The tight connection to the modulus of continuity they define is very nice, and so is the super-efficiency result. I think this is a very interesting new direction which may yield some nice algorithmic ideas in the future.  The paper does have some drawbacks. In my opinion, the algorithms presented are not strong, although the framework is very interesting. The 1D algorithm is in my opinion not vey satisfying, and neither is the information theoretic upper bound presented. The latter is particularly worrisome because it suggests to me that perhaps this notion of local minimax complexity is not quite the right thing to do, since it allows these algorithms which cheat so much. The minimax setting itself is a bit strange, because the noise is assumed to be Gaussian, whereas usually it suffices to have a second moment condition on the gradient oracle.


Fast and accurate spike sorting of high-channel count probes with KiloSort
New silicon technology is enabling large-scale electrophysiological recordings in vivo from hundreds to thousands of channels. Interpreting these recordings requires scalable and accurate automated methods for spike sorting, which should minimize the time required for manual curation of the results. Here we introduce KiloSort, a new integrated spike sorting framework that uses template matching both during spike detection and during spike clustering. KiloSort models the electrical voltage as a sum of template waveforms triggered on the spike times, which allows overlapping spikes to be identified and resolved. Unlike previous algorithms that compress the data with PCA, KiloSort operates on the raw data which allows it to construct a more accurate model of the waveforms. Processing times are faster than in previous algorithms thanks to batch-based optimization on GPUs. We compare KiloSort to an established algorithm and show favorable performance, at much reduced processing times. A novel post-clustering merging step based on the continuity of the templates further reduced substantially the number of manual operations required on this data, for the neurons with nearzero error rates, paving the way for fully automated spike sorting of multichannel electrode recordings.

The authors develop a new spike sorting method. Unlike other approaches, their method is not based on dimensionality reduction. Their GPU based implementation  leads to dramatic time savings as compared to KlustaKwik. Importantly, their method utilizes a novel template-wise smoothing method based on the SVD decomposition of the cluster average of the traces (templates), which is the key for obtaining the aforementioned computational gains.
Improvements in spike sorting are always appreciated because of the always-increasing electrode array capabilities (e.g number of electrodes or density) leading to new computational challenges. Methodologically, their contribution is the observation that one can take the SVD of the mean of raw traces in the cluster order to i)regularize shape of templates, ii)obtain computational gains via a GPU implementation of the operations in equation 6. This methodological contribution leads to impressive improvements in time while maintaining comparable levels of accuracy. In general it is a very nice paper.  However, I think this paper has major issues and it is not suitable for publication in the NIPS conference/proceedings. 1)Overall, it gives the impression that it hasn't been fleshed out enough. We see this in i)missing references due (I suppose) errors in compilation of the pdf ii)lack of scales and labels in graphs iii)lack of enough references (beyond the ones that are missing in i). See for example section 2.1 or references supporting their optimization method) iiii)the authors don't do a very good job in graphically displaying their results/methods. 2)Their methodological contribution may be a bit incremental for a NIPS paper. It is certainly smart the SVD trick they apply, and how it entails computational gains, but the rest of the innovations are rather simple algebra tricks and don't lead to the same impressive results.  3)The way in which the authors present their method in relation to already published literature is not quite accurate: for example, in lines 63-66 it is stated that they improve the method in [1] but the motivation in [1] is actually orthogonal: they utilize continuous time modeling as a device to resolve overlapping spikes that are separated by a sub-sampling-rate time-lapse. Then, it is imprecise to call this an improvement over [1], as the author, I believe, are instead borrowing the idea from [1]  and applying it to a simplified context (which ends up being useful, of course). Also, the authors state that their non-PCA approach is an innovation (line 68-70 'unlike previous approaches'). However, this has already became a common practice in the spike sorting literature ([1],[2],[3],[4]).  4)The time comparison the authors present is questionable (lines 220-225). Why didn't they attempt a GPU against GPU comparison?  As a summary, it has a very nice key insight that could be the seed of a very nice work. However, the snapshot presented by the authors at this time seems still too "under-development".   [1]Chaitanya Ekanadham, Daniel Tranchina, and Eero P Simoncelli. A unified framework and method for 296 automatic neural spike identification. Journal of neuroscience methods, 222:47–55, 2014. [2]Jonathan W Pillow, Jonathon Shlens, EJ Chichilnisky, and Eero P Simoncelli. A model-based spike 293 sorting algorithm for removing correlation artifacts in multi-neuron recordings. PloS one, 8(5):e62123, 294 2013. [3]D. Carlson, V. Rao, J. Vogelstein, and L. Carin. Real-Time Inference for a Gamma Process Model of NeuralSpiking.NIPS, 2013 [4].D. Carlson, J. Vogelstein, Q. Wu, W. Lian, M. Zhou, C. Stoetzner, D. Kipke, D. Weber, D. Dunson, and L. Carin.Multichannel Electrophysiological Spike Sorting via Joint Dictionary Learning & Mixture Modeling.IEEETBME, 2014


Sublinear Time Orthogonal Tensor Decomposition *
A recent work (Wang et. al., NIPS 2015)  gives the fastest known algorithms for orthogonal tensor decomposition with provable guarantees. Their algorithm is based on computing sketches of the input tensor, which requires reading the entire input. We show in a number of cases one can achieve the same theoretical guarantees in sublinear time, i.e., even without reading most of the input tensor. Instead of using sketches to estimate inner products in tensor decomposition algorithms, we use importance sampling. To achieve sublinear time, we need to know the norms of tensor slices, and we show how to do this in a number of important cases. For symmetric tensors T = k i=1 λ i u ⊗p i with λ i > 0 for all i, we estimate such norms in sublinear time whenever p is even. For the important case of p = 3 and small values of k, we can also estimate such norms. For asymmetric tensors sublinear time is not possible in general, but we show if the tensor slice norms are just slightly below T F then sublinear time is again possible. One of the main strengths of our work is empirical -in a number of cases our algorithm is orders of magnitude faster than existing methods with the same accuracy.

This paper presents a randomized method for decomposing a symmetric, orthogonal tensor that is “nearly low rank”. Specifically, the tensor should expressible as a low-rank tensor + an arbitrary noise tensor with bounded norm. The fact that the tensor is symmetric and composed of orthogonal components may sound restrictive, but this is exactly the sort of tensor decomposition problem that comes up in spectral methods that solve Latent Dirichlet Allocation problems. So, it’s an interesting model to study.  The method introduced very closely follows work in a paper from NIPS 2015, “Fast and guaranteed tensor decomposition via sketching” which uses a standard tensor power method for decomposition, but speeds up each iteration by accelerating the required tensor contractions (vector dot products against the tensor) via randomized sketching techniques. This paper uses the same algorithm, but instead of using sketches based on “random projections”, which take random linear combinations of all of the components of the tensor, it uses a subsampling technique to approximate the contractions. The authors main selling point is that this approach should allow for better running times since they don’t have to touch every entry in the tensor with each iteration.  Sampling also avoids and expensive preprocessing cost incurred by the sketching algorithm.  Without knowing the prior work, it’s a bit tough to understand how the runtime of this algorithm compares. In the introduction it’s stated that the sampling algorithm is slower per iteration than the sketching methods, which are accelerated with sparse hashing and fast Fourier methods. However, runtimes are stated in terms of the sketching dimension b which is not the same for both algorithms. It can vary by orders of magnitude in the experiments. It’s stated that the algorithm has a faster runtime overall, I believe because of preprocessing costs in the sketching algorithm, but I would like to see these costs laid out so that the overall runtime comparison is clear.  Regardless, the algorithm performs well experimentally, providing a (3x-6x) speedup for small datasets and up to a 50x speedup for large synthetic data sets. This is especially impressive since the sampling algorithm is much simpler than the prior work, whose implementation seemingly requires many optimizations and detailed parameter setting.
This paper is decently written and the experimental results look promising. However, I don’t think the result is quite strong enough that I would recommend acceptance for NIPS. In general, the paper leans heavily on the work in “Fast and guaranteed tensor decomposition via sketching”. The idea to use sampling is nice, but the analysis is straightforward since the sampling scheme plugs into the prior work black-box. The authors do not prove any tighter theoretical guarantees and it’s not clear if their methods give any provable runtime improvements over the sketching methods. I would encourage the authors to clarify this aspect of their paper.    More generally, while it’s nice to have theoretical guarantees, I worry that the bound in Lemma 10 is too complex/loose to provide indication of the algorithm’s quality. In addition to a very strong requirement on the noise eps, it has an inherent n^3 dependence, a stable rank dependence, a polynomial dependence on the failure probability, and spectral gap and condition number dependencies that I don’t expect to be small. It would be great to see if the analysis of these sketching techniques can be pushed beyond the work in “Fast and guaranteed tensor decomposition via sketching” to obtain a more interpretable/convincing bound. In light of it’s demonstrated practical performance, perhaps sampling can be analyzed in a tighter way? As a more specific point, I don’t understand the meaning of r(λ)=max(λ_i/λ_j). Wouldn’t this simply equal λ_1/λ_k?  As far as presentation goes, I have a couple of comments:  I think it’s a bit confusing to claim that the algorithm runs in “sublinear time”: it seems that the final runtime in Theorem 10 has a total O(n^3) dependence since b depends on n^2. I understand that that in practice the running time *can* be sublinear because the required number of samples is much lower than predicted by the theory, but I think this should be made more clear in the abstract and introduction.   For a shortened form of the paper (like the NIPS submission) I would prefer to see the sampling analysis (currently in the supplemental material) in the main text, in lieu of the discussion of how to very efficiently generate importance samples. While I appreciate the careful work on this aspect of the algorithm (the actually sampling process is often overlooked by other papers!) ultimately it’s focusing on smallish log factors and is somewhat orthogonal to the main message of the paper. It could be relegated to an appendix.


Semiparametric Differential Graph Models
In many cases of network analysis, it is more attractive to study how a network varies under different conditions than an individual static network. We propose a novel graphical model, namely Latent Differential Graph Model, where the networks under two different conditions are represented by two semiparametric elliptical distributions respectively, and the variation of these two networks (i.e., differential graph) is characterized by the difference between their latent precision matrices. We propose an estimator for the differential graph based on quasi likelihood maximization with nonconvex regularization. We show that our estimator attains a faster statistical rate in parameter estimation than the state-of-the-art methods, and enjoys the oracle property under mild conditions. Thorough experiments on both synthetic and real world data support our theory.

The authors present a new approach to comparing pairs of graph structures. In contrast to previously developed approaches, the proposed approach 1) estimates the difference-graph directly and in a sparse way, thus enabling a more sparse representation of the difference graph (as opposed to estimation of the two graphs first then comparing), 2) the proposed approach relaxes the Gaussian assumption typically made by other methods by using transelliptical distributions, and 3) it uses a family of non-convex penalty functions that allow efficient inference.
The article is well organized and the results are very thoroughly presented (over 10 pages of proofs).    - In the introduction, the authors state estimation of two graphs first, then comparison to estimate the difference, requires estimating twice as many parameters (and therefore twice as many observations) to ensure estimation accuracy. Is this true for the class of methods that regularize the difference between two graphs (e.g. penalize differences in edge weights between two graphs)?  - I find Assumption 4.2 to be quite strong and unrealistic – can we expect that the number of non-zero entries in the difference-matrix to not depend on the dimensionality of the data (d)? E.g. Would we expect the number of non-zero entries between graphs of dimension 100 to be on the same order as differences between graphs of 1,000,000 dimensions?  - It would be nice if the authors provide some intuition for the significance of Assumption 4.3.  - It was unclear to me how sparse the matrices are that were generated from simulations in section 5.1. Also, what is the motivation behind the choice of functions f (line 230)?  - The simulations in 5.1 need to explore performance with respect to different sparsity patterns of \Sigma_x, \Sigma_y and delta, since the proposed method is geared towards sparse estimation of the difference-graph. Also, larger graphs (e.g. 1,000 nodes, 10,000 nodes) would be useful as well (since the proposed approach is more likely to be used on high dimensional data).  - I found the real-world data application of 5.2 unconvincing for the following reason. The pathway TGFB was hand-chosen, likely because of its extremely high importance in cancer. However, first the validation involved verifying genes involved in differential edges are important in the same kind of cancer. TGF-B is heavily implicated in many, many cancers, and so you would expect many of the genes in that pathway a priori to be involved in cancer to begin with (without even looking at the graph). Second, the validation is of the genes (the nodes), not of the differential edges, which is what the method is inferring. Third, it is well known that hubs in a network tend to be more important (and also more likely to be co-opted by cancer). Hubs in either graph (or both) are also more likely to have a differential edge detected by the method (you can’t have a differential edge adjacent to a node if that node does not have any edge in either graph). So, the method is more likely to pull down a hub node by chance anyways (e.g. a competitive method that just identifies nodes based on degrees in either network may be just as good at pulling down relevant genes). Fourth, there is no mention of false-negative rate (how many ovarian cancer genes were missed by the method/did not have a differential edge). Better validation would have been to identify some genes that are unique to the subtypes studied and looked at their recovery.  - Figure 3 was also unconvincing for the same reason.  - Real world data is noisy and messy.  It is important to record how the networks change for different bootstraps of the samples, or at least the variance for the 5 different 5-fold cross-validations at the optimal values of the tuning parameters of \lambda and b. (The optimal values of \lambda and b should be stated as well.)   - The interpretation of the differential networks is unclear, as the edges between genes in the differential networks do not match direct interactions in the KEGG pathways and their relationships are not discussed in the article.  Is the motivation simply to find individual genes which are differentially expressed? It is also unclear why the gene CSF2RB found by SepGlasso (p. 8, line 277-9) is “wrong” rather than just “different”. (Searching the NCBI entry for this gene gives a reference to an article on colon cancer.)  - Has the data been normalized (i.e. assumed to be Gaussian), or modeled using a transelliptical distribution (as discussed on p.1, line 21-24)?   Are the assumptions on the data satisfied, or if not (d > n?), why is it justifiable to use the method on this dataset?  - The results for LDGM-MCP seem to recapitulate those from the DPM reference [32]. However, only one gene (e.g. THBS2 rather than COMP and THBS2 for the TGF-beta pathway) appear in the results for DPM in the manuscript, whereas both appear in [32].  Why is there a discrepancy?  - In searching the KEGG Apoptosis pathway from (http://www.genome.jp/dbget-bin/www_bget?hsa04210), I could not find gene PRKAR2B or its Entrez gene ID 5577 on the page.   - Given that in the ovarian cancer dataset both n and d are small (approx 100), the two precision matrices and their difference could be calculated in this case, for comparison with LDGM-MCP.  (The only advantage to estimating the differential network in the manuscript is said to be the factor of 2 in the number of parameters to be estimated, p.2 lines 43-44 and lines 73-74). However, I was wondering if there also an improvement in stability when only the difference is calculated?  - I have not seen the Kendall tau statistic used estimations of sample covariance of application papers involving gene expression profiles.  Its use seems necessary in the proofs (Lemma C.1), and this should be stated if it is indeed the case.  - It would be very useful to provide an implementation of the method (or even better, the source code) to ensure transparency and reproducibility of the results.   - There are some minor typos and errors in language/flow of writing.  Some of them are listed below. “S = supp(\Delta^*)” appears on p.4, line 122 but “supp” is only defined on p.4, line 141.  The definition could be included in the “Notation” section at the end of the Introduction, which was useful.  “theories” should be replaced with “theorems” (e.g. p.4 line 141, p.5, line 174)  p.4, line 156, “||\Delta||^*_{0,0}” should be replaced by “||\Delta^*||_{0,0}” p. 6, line 227: “by the huge package” (or “by the huge R package” to specify that it is in R), and the version used should also be mentioned, for reproducibility p. 6, line 232, “mdoels” to be replaced by “models”


Unsupervised Learning of 3D Structure from Images
A key goal of computer vision is to recover the underlying 3D structure that gives rise to 2D observations of the world. If endowed with 3D understanding, agents can abstract away from the complexity of the rendering process to form stable, disentangled representations of scene elements. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained jointly, end-to-end, and directly from 2D images without any use of ground-truth 3D labels. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.

This paper provides a generative model of 3D volumes and 2D images using an internal 3D latent representation. The model can be conditioned on a context, which in turn can be either a category or multiple views of the 3D volume. In a very similar way to DRAW, the model transforms a set of latent Gaussian variables into an internal 3D representation, which in turn is converted to a 2D image via a convolutional network or OpenGL renderer. Inference in this generative model is performed using variational inference with a recognition network. Five different tasks are considered.
The paper is well written, clear and contains appropriate references. Also the number of figures help understand and illustrate the text appropriately.  The technical content of the paper seems sound, but its ideas are not particularly novel. The proposed model is an extension of DRAW, which now includes a context (which is observed and used as an additional input to the recognition and generation networks) and generates 3D representations instead of 2D representations. This requires the addition of a projection step to 2D at the end of the generation process when the observations are 2D. Well-known variational inference is applied to the resulting model.  Some of the claims in the introduction are misleading. "We show how the aforementioned models and  inference networks can be trained end-to-end directly from 2D images without any use of ground-truth 3D labels.". This make it seem as if 3D latent structure was discovered exclusively from 2D information. This is not the case, and prior knowledge about 3D structure and projection from 3D to 2D is included in every version of the model. Only the last experiment seems not to use 3D input data directly (the multi-view experiment uses depth in the provided context). But in that experiment, an OpenGL renderer is introduced in the loop, providing, if we wish, an infinite amount of 3D ground truth by sampling from its input-output pairs. Additionally, not every 3D shape can be accommodated by the model of the last experiment due to its particular parameterization, which is not pointed out in the paper.  I would like more clarity regarding training and test splits. For instance, the objects in Figures 9 and 10 were observed at all during training or not? I would like to distinguish generalization to new perspectives from generalization to entirely new shapes.  One thing I found concerning about this paper is the lack of comparison with benchmarks. The experiments comprise 5 different tasks. I would like to see the performance of a reasonable benchmark for each of these tasks, so that it is possible to judge whether the results produced by this model are particularly impressive or not.  - For instance, Section 3.5 discusses multiview contexts. The text is not clear about the nature of the views, but the corresponding figure and legend clarify that the input views are 3D. It seems like it should be relatively easy to infer a 3D representation from multiple views with depth information. The obtained results should be compared with those of a different technique for the same task to ascertain whether the model is working particularly well or not.  - Similarly Section 3.6 uses a very specific parameterization of the input space that restricts the applicability of the model. My understanding is that the same experiment could not be performed with the objects from ShapeNet, for instance. This should be pointed out. Are the objects we see in Figure 10 used to train the model? If they are as I suspect, couldn't the model be basically memorizing 3D structures for each object and then inferring pose for each test image? (as opposed to being able to get 3D understanding of a new object). Then the problem could be posed as, given a set of training images, identify same-object clusters and find its 3D representation. A baseline for this task should be included so that the quality of the proposed model could be judged.  - Section 3.3 and Figure 7: Again the generalization capabilities are unclear. Is the model generating _new_ shapes within the same category or multiple rotations of already-observed shapes? The nearest training neighbor to each generation, computed using a meaningful metric (that matches shape regardless of rotation, translation, and scaling) could be useful here.  Minor:    Section A.3: "of the three kernels": there are only 2 kernels in the 2D ST. Section 2.1:  "fwrite(st,h_t-1;theta_w) = VST(g1(st),g2(st))" This equation doesn't seem correct, isn't h_t-1 used during writing?


Dueling Bandits: Beyond Condorcet Winners to General Tournament Solutions
Recent work on deriving O(log T ) anytime regret bounds for stochastic dueling bandit problems has considered mostly Condorcet winners, which do not always exist, and more recently, winners defined by the Copeland set, which do always exist. In this work, we consider a broad notion of winners defined by tournament solutions in social choice theory, which include the Copeland set as a special case but also include several other notions of winners such as the top cycle, uncovered set, and Banks set, and which, like the Copeland set, always exist. We develop a family of UCB-style dueling bandit algorithms for such general tournament solutions, and show O(log T ) anytime regret bounds for them. Experiments confirm the ability of our algorithms to achieve low regret relative to the target winning set of interest.

This paper deals with the dueling bandit problem in situations where the goal is to find other notions of winners than the Condorcet one (which is not guaranteed to exist) or the Copeland one (which does always exist, but has certain flaws). Unlike the MAB problem, the question of what constitutes the best arm(s) is not completely straightforward to answer in the dueling setting, in particular when there is no single arm that is preferred to all other arms (aka the Condorcet winner).  Indeed, the field of social choice theory has devised a near infinite supply of such notions of winners: the reason for this overabundance of definitions is that in the absence of a Condorcet winner, it is difficult to find a definition that satisfies all reasonable properties that one might hope for, so various definitions have been proposed that satisfy various sets of desirable properties. Given that, it is natural to seek algorithms that can find these other winners, since depending on the application domain, it might very well be that the Condorcet winner does not exist and that the Copeland winner is undesirable.  With this in mind, the authors propose a flexible algorithm that is capable of accommodating such needs, given certain rather weak assumptions on the sought after notion of winner. In plain terms, the algorithm can find solution concepts that are stable with respect to optimistic estimates of the underlying preference matrix. This result can be seen as a refinement of the ideas in reference [3] in the paper. The main difference is that while the SAVAGE algorithm in [3] was restricted to the finite horizon setting, the algorithm in this paper does not need knowledge of the horizon of the experiment in advance.  Moreover, the authors conduct experiments on three examples comparing their algorithm adapted to three specific notions of winners against three existing algorithms, although the description of the experiments is lacking in some important details. First of all, it is unclear whether the comparison is being carried out against the Condorcet or the Copeland version of SAVAGE, and if the former is the case, why was there no comparison against Copeland SAVAGE? Secondly, I do not understand how RUCB could have sublinear regret on the Sushi and MSLR examples, since it crucially assumes the existence of a Condorcet winner, which is absent in those examples. Finally, it is rather disappointing that the authors do not compare against the state of the art duleing bandit algorithm(s) in the Condorcet case, proposed in reference [9] in the paper.
Overall, the theoretical/algorithmic portions of the paper are written reasonably clearly and the arguments seem correct to me. On the issue of novelty, the domain into which the paper is venturing is relatively novel, but the proofs techniques are not extremely original, although there is definitely value in using existing ideas to solve a new problem.  There are also some issues with the experiments (as I mentioned above), so I urge the authors to revisit that portion of the paper.


Hardness of Online Sleeping Combinatorial Optimization Problems
We show that several online combinatorial optimization problems that admit efficient no-regret algorithms become computationally hard in the sleeping setting where a subset of actions becomes unavailable in each round. Specifically, we show that the sleeping versions of these problems are at least as hard as PAC learning DNF expressions, a long standing open problem. We show hardness for the sleeping versions of ONLINE SHORTEST PATHS, ONLINE MINIMUM SPANNING TREE, ONLINE k-SUBSETS, ONLINE k-TRUNCATED PERMUTATIONS, ONLINE MINIMUM CUT, and ONLINE BIPARTITE MATCHING. The hardness result for the sleeping version of the Online Shortest Paths problem resolves an open problem presented at COLT 2015 [Koolen et al., 2015].

The paper solves an open problem presented at COLT 2015 [Koolen et. al. 2015] and proves the hardness of Online Sabotaged Shortest Path problem with per-action regret, and furthermore, establishes a hardness results for a general category of Online Sleeping Combinatorial Optimization (OSCO) Problems. It defines OSCO problems as online combinatorial optimization problems in sleeping setting in which in each trial a subset of actions become unavailable due to unavailability of a subset of elements. Basically, the paper shows that OSCO problems are as hard as PAC-Learning of DNF expressions -- which is a long standing open problem.  In a nutshell, the hardness result is shown as follows: the paper reduces the problem of "Online Agnostic Learnign of Disjuction" into "Hard Instances" of OSCO problems with per-action regret via a proof that has similarities to a proof in [Kanade and Steinke, 2014]. Incorporating the results of related works like [Kanade and Steinke, 2014], [Kearns et. al. 1994], [Kalai et. al. 2012], the paper mentions that "Online Agnostic Learning of Disjuction" is at least as hard as "Agnostic Improper PAC Learning of Disjuctions" which is also at least as hard as "PAC-Learning of DNF expression". Interestingly enough, because of the online-to-batch conversion argument in [Kanade and Steinke, 2014], the hardness results seem to be also true for a benign form of adversary namely stochastic availabilities and loss (i.e. they are drawn from an unknown but fixed joint distribution).  After proving the general hardness results, the paper provides "Hard Instances" of various well-known OSCO problems to establish their hardness in particularv -- including Online Sabotaged Shortest Path.
The main proof of the general hardness result is clearly explained and looks sound and correct. The idea of the proof is similar to [Kanade, Steike 2014] and therefore not quite ground-breaking, but notably novel. Other technicalities of the paper are also well written to some extent; however, there are still some confusing parts. Concretely, section 3 is not straight-forward to follow. For example, in the first paragraph the difference between "Agnostic Improper PAC-learning of disjuction " vs "PAC-learning of DNF Expressions" is not clear -- perhaps mainly because there is a discrepency between the naming in the paper and the related work [Kalai et. al. 2012] in which the term "Reliable Agnostic Learning" is used instead. Also the reviewer could not see the hardness result even in iid case using [Kanade Steinke 2014] (it does not seem to be clearly mentioned in cited paper).  Generally, the paper has a good flow and it is well-organized. The presentation is quite readable and understandable. The reviewer believes the impact is significat since it not only solves an open problem presented in COLT 2015, but also establishes a more general hardness for a category of problems with a loose form of adversarial setting.


Mapping Estimation for Discrete Optimal Transport
We are interested in the computation of the transport map of an Optimal Transport problem. Most of the computational approaches of Optimal Transport use the Kantorovich relaxation of the problem to learn a probabilistic coupling γ but do not address the problem of learning the underlying transport map T linked to the original Monge problem. Consequently, it lowers the potential usage of such methods in contexts where out-of-samples computations are mandatory. In this paper we propose a new way to jointly learn the coupling and an approximation of the transport map. We use a jointly convex formulation which can be efficiently optimized. Additionally, jointly learning the coupling and the transport map allows to smooth the result of the Optimal Transport and generalize it to out-of-samples examples. Empirically, we show the interest and the relevance of our method in two tasks: domain adaptation and image editing.

The authors study the optimal transport problem, motivated by the desire to come up with a model which better generalizes to unseen examples. Standard techniques do not generalize as the transport map is not defined for new data and needs to be recomputed whenever new data point is obtained. To this effect, the authors propose an optimization model which enables the joint learning of the transport map and the probabilistic coupling. The (visual) advantages of the new model are validated on domain adaptation and image editing (face swapping) problems. The objective/loss function in new model is, by design, jointly convex in the two variables (transport and coupling), and an alternating minimization method is employed to solve it.
1) The paper reads well and the new model for joint optimization of transport and coupling looks promising. The computational approach satisfactorily demonstrate the usefulness of the new model and the solution approach.  2) The authors compare to the work of Perez  et al [28] which dates back to 2003. Was there no progress in the field in the last 13 years? A detailed commentary on why comparison is done against [28] and why not against newer approaches is needed. 3) It should be explained in words what (1) means, also explaining each term appearing in (1).  The current text does not explain the meaning well. It was not explained what notation # means; all used notation should be explained for the paper to be accessible to a wider audience.  4) What is {\cal P} in Line 75? This was not explained – notation is used without any word of warning or advice. 5) Equation (4) should be explained in more detail to a lay audience. What is being mapped to what? Are we mapping index i to   x\in \Omega? What is \Omega (this was never defined)?  6) Line 93: Why do you focus on uniform samplings? What do you mean by uniform samplings? This should be explained at more length as it seems to be a critical assumption/limitation of focus. 7) The choice and meaning of the various constants appearing in the new model (6) is not discussed. For instance, why should one normalize by max(C)? After all, lambda_\gamma can account for this. Why should one normalize the regularizer by d^2? Again, \lambda_T can absorb this term. How about the factor of 1/d in front of the first term?  8) On lines 116-118 it is claimed that since the loss in (6) is jointly convex, alternating optimization will converge to the optimal solution. First of all, you need to prove that an optimal solution exists (first argue boundedness below by 0, and so on). Second, even if optimal solution exists, alternating optimization (or block coordinate descent with 2 blocks) need not converge. A simple example of this is min max{|x_1|, |x_2|}. Starting from  x=(x_1,x_2) = (1,1), no coordinate direction leads to descent. Hence, the method can get stuck at a non-optimal point. This illustrates that additional conditions to convexity are needed for block coordinate descent to work. Are these satisfied for this function? The paper would be much stronger if this was established. 9) I do not understand the assumption on H in Theorem 1. The assumption seems to be a definition of convex combination of 2 functions. Hence, it is not an assumption! What should be assumed is that H is itself convex. 10) The only theoretical result in the paper, Theorem 1, is a trivial observation.  The paper contains a number of typos and English usage issues; some of which are listed below. The authors should re-read the paper and do corrections. In general though, the paper is well written.  1) Line 29: bound -> bounds 2) Line 31: Most of -> Most 3) Line 36: from -> for 4) Line 37: have -> find 5) Line 47: to an -> to 6) Line 51: easily kernelized that -> easily kernelized, which 7) Line 53: evidence on -> evidence for 8) Line 54: interest -> usefulness 9) Line 54: on domain -> in domain 10) Line 56: remaining -> rest 11) Line 71: assuming for -> assuming 12) Line 73: of the -> of 13) Eq (2) should end with a full stop 14) Line 77: writing -> formulation 15) Eq (3) should end with a comma 16) Line 86: computed ->computed, 17) Eq (4) should end with a comma 18) Eq (4): Presumably, \Omega should be replaced by \Omega_{\cal S} 19) Line 93: an -> a 20) Line 93: hence -> , whence 21) Eq (6): Defined max(C) – is this the maximal element in matrix C? This was not defined 22) Eq (6) should end with a comma 23) Line 111: the -> The 24) Line 114: a block -> block 25) Line 114: [16] where -> [16], where 26) Line 122: block coordinates approach -> block coordinate descent approach 27) Line 125: problems -> problem  I stopped checking for typos beyond this – but I expect the same frequency in the rest of the paper. Please correct them all.  I just did a very cursory check beyond this in the Supplementary: 1) Line 3: Theorem -> theorem 2) Line 9: closeness -> proximity 3) Second inequality after line 14 should be an equality  Bibliography:  [5, 6, 14, 21, 27]: wasserstein -> Wasserstein [7]: bregman -> Bregman [16]: capitalize journal name [19]: frank-wolfe -> Frank-Wolfe  Similar issues appear in biblio for the supplementary part.  ---- post-rebuttal remarks -----  I have read the other reviews and the rebuttal and as a result keep my scores unchanged.


Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game -'Montezuma's Revenge'.

The paper addresses deep hierarchical RL with a two-levels approach. Low-level controllers are used to reach goals, and a higher-level controller chooses the current goal. Goal specification is not clearly described. The model is tested first in a toy problem and then on Montezuma's revenge, a hard ATARI game. Results are preliminary, but quite promising.
The paper addresses the timely question of building a hierarchy of behaviours out of recent deep reinforcement learning methods. This is one of the hot challenges at the moment and several teams are on that (actually, the arXiv version of the paper is already cited 10 times). The paper proposes a promising contribution to the domain, but my general feeling is that the paper and the corresponding work are not mature enough for publication at NIPS: several points need to be clarified and the empirical evaluation is too preliminary.  Globally, the major issue in the field is the automatic extraction of a hierarchy from data. Here, a good deal of the hierarchy building process is left to the engineer through the identification of a set of goals. In itself, this is not a major flaw of the paper, but the limitations of the work should be discussed under this light.  About the literature review section, two points:   1°) "Our approach does not use separate Q-functions for each option, but instead treats the option as part of the input, similar to [18]. This has two advantages: (1) there is shared learning between different options, and (2) the model is potentially scalable to a large number of options."  => The statement about these two advantages should be supported either by a reference or an empirical study, where the authors would compare both approaches  2)° In the "Intrinsically motivated RL" and the "Cognitive Science and Neuroscience" subsections, nothing is said to relate the described works to the submitted paper. In particular, for the latter, the content looks quite far away. Also, the "Deep Reinforcement Learning" subsection does not bring much. So my general feeling is that this section should be more targeted to recent deep hierarchical RL work, and should save space for additional studies such as the one described in point 1 or improving the model description (see below).  By the way, given that the arXiv version of the paper was published a while before NIPS, the authors should refresh their literature review section the many recent works in the domain. E.g. Model-free episodic control, several works about macro-actions, using A3C instead of DQN (fine if that's left for future work ;)), etc.  About the model description:  The model section could be much clearer. Although the right-hand side of Fig.1 shows it, it is not obvious from the notations in the text that the controller networks take s and g as input and a as output, and that the "meta-controller" takes s as input and g as output. It would be clear if you write : Q1(A|s,g) and Q2(g|s), for instance.  Also, there should be a module (inside the meta-controller?) checking that a goal is reached. This is not described clearly. The language to specify goals is not described either (despite the connection to Duik's work about object-oriented MDPs).  The replay buffer initialization is not described.  The exploration strategy deserves its own subsection. Using a separate epsilon for each goal should be discussed. Also, the adaptive annealing method for \epsilon_1,g is not described, though this is not standard.  It should be specified that a set of goals is given as input to the algorithm. Also, the method for checking that a goal is reached should be present in the algorithm. This part is poorly specified and looks quite ad hoc (see below about Experiment 2).  About the study and results:  In Experiment 1, the structure of the controllers and meta-controller are not described (and should be). It is only specified that it is not a deep network.  In Experiment 2, the lack of a clear specification of how the goals are managed is a major issue. How many entities do you have in Montezuma's revenge? If you have two instances of the same entity, do they correspond to two different goals? Is reaching an entity with the agent the only kind of goal? What about avoiding monsters at pacMan, for instance? Even if what the authors did is completely ad hoc to a specific game, it should at least be described accurately.  The parametrization of the architecture and learning process looks quite arbitrary and should be discussed.  Finally, the small paragraph about missing ingredients convinces the reader that this work, although an interesting starting point, is not mature enough for publication in a major conference, particularly one that is biased towards clean theoretically-grounded work.