Memory-Efficient Backpropagation Through Time
We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs). Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation. The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost. Computational devices have limited memory capacity and maximizing a computational performance given a fixed memory budget is a practical use-case. We provide asymptotic computational upper bounds for various regimes. The algorithm is particularly effective for long sequences. For sequences of length 1000, our algorithm saves 95% of memory usage while using only one third more time per iteration than the standard BPTT.

Training recurrent neural networks requires unfolding the network in time and computing the backpropagation "through time". In order to perform it efficiently and to avoid recomputing intermediate values, these are usually stored in memory. However, as considered sequences increase in length and networks grow in size, the amount of available memory might not be sufficient. An usual solution is to truncate the backpropagation, resulting in approximate weight updates. It is also possible to recompute the forward pass, which is inefficient in computation time. Finally, heuristical "divide-and-conquer" approaches have been proposed to trade off between computation time and memory efficiency. The authors of this paper propose a new solution, in order to find the optimal strategy to get an algorithm both memory-efficient and not too expensive in terms of computation. More particularly, they define a cost as the number of forward operations needed to compute BPTT given a fixed memory budget. This cost is minimized with dynamic programing, in terms of when to store the intermediate results in memory versus when to recompute them. They study storing the hidden state of recurrent networks (i.e. output of the recurrent neurons), the internal state (needed to compute this output) or both, which each correspond to a different formulation of the cost. They show that this simple method can achieves 95% memory saving while being only 33% slower, and derive theoretical bounds in terms of the sequence length and memory budget. Their method is compared to naive baselines and to an handcrafted heuristical method.
This paper is very well written and pleasant to read. Although the figures supporting the explanations take some room, every bit is important and adds to the clarity of the paper.  The idea to use dynamic programing to define when to store the intermediates in order to limit the number of forward operations in fixed memory budget scenarii is quite simple and well defined. The notation is clear and the figures are helpful. The chosen baselines are relevant (limit cases + heuristical divide-and-conquer method previously published). More importantly, this paper addresses a real problem, given the popularity of recurrent neural networks these days, along with the increasing size of models designed and sequences processed. I believe that in that scenario, many experimenters give up exact computation and use a truncated version of the backpropagation through time (quite a few papers in the last few years). This paper provides a simple and elegant solution, with all necessary details to implement it, including pseudo-code, which is highly appreciable. Following are a few remarks and cosmetic issues:  - I did not quite understand how the 33% time increase can be deduced from Fig.5a. (end of p4)  - Sect. 3.2 : typo in second paragraph ( "on less memory slot" )  - Fig 2b (in caption) : "per time step. sured in ms"  - Sect. 4.2 : missing cap "in figure 7" (last paragraph)  - Suppl., p3, last line : pb with ref: "Section ??"  - Right-hand side of Fig.7 I guess the point was to show that curves overlap, but one can hardly see those curves. More generally, a legend on the figures rather than inside the captions would be appreciated, and increase the clarity of the figures. The titles and axes labels could also be bigger.


Fast and Provably Good Seedings for k-Means
Seeding -the task of finding initial cluster centers -is critical in obtaining highquality clusterings for k-Means. However, k-means++ seeding, the state of the art algorithm, does not scale well to massive datasets as it is inherently sequential and requires k full passes through the data. It was recently shown that Markov chain Monte Carlo sampling can be used to efficiently approximate the seeding step of k-means++. However, this result requires assumptions on the data generating distribution. We propose a simple yet fast seeding algorithm that produces provably good clusterings even without assumptions on the data. Our analysis shows that the algorithm allows for a favourable trade-off between solution quality and computational cost, speeding up k-means++ seeding by up to several orders of magnitude. We validate our theoretical results in extensive experiments on a variety of real-world data sets.

The paper proposed a MCMC approach to approximate the D^2-sampling (k-means++) for finding k centers in clustering, by modifying the proposal distribution of an earlier paper by Banchem 16. The algorithm is able to approximately maintain the O(log k) approximation guarantee to the k-means objective for any dataset (as is k-means++), if their analysis is correct. The running time of the algorithm is linear in the number of data size, but shaves off a factor of k (the number of clusters) comparing to that of k-means++.
Technical quality: I have two doubts regarding the proof of their Lemma 2:  1. The authors argue, in the case of \phi_{C}(X) < \epsilon_1\phi_{c_1}, the claim holds trivially. I don't see how this goes through. First, I don't see why A^{c_1}(C,l) < =\phi_{C}(X). As I understand it, A^{c_1}(C,l) is the expected cost of C while \phi_{C}(X) is the actual cost of C (random quantity). If so, why is the former upper bounded by the latter? Second, even if the relation above holds, I don't see how the statement holds, since then we'll get  A^{c_1}(C,l) < =\phi_{C}(X) < \epsilon_1\phi_{c_1}(X). How does this imply the statement using P^{c_1}(C,l) < 1?  2. I don't see how inequality (8) holds, since I can't see why the equation right above it holds, using the definition of \pi.   Novelty: I think the idea of using MCMC methods to approximate D^2-sampling scheme is great. But since this paper is not the first to propose this approach, and the theoretical analysis seems incremental to that of Banchem 16, I think I should take some points off here.  Potential impact or usefulness: I think this paper could potentially have good impact in large-scale clustering applications, given that k-means++ is widely used together with Lloyd's algorithm for clustering. Although the proposed algorithm still has a linear running time dependence on the data size, I think the fact that they can shave off a factor of k while approximately maintaining the performance guarantee of k-means++ could mean a lot in practice. The proposed algorithm also seems simple enough to be implemented in practice, comparing to other scaled versions of k-means++. In general, I think this research direction worths further exploration.  Clarity and presentation: In terms of highlighting their contribution to previous work, the paper does a good job via both writing and experiments. In terms of the analysis, I think the authors should make more efforts in clarifying definitions and providing For example, the definitions of \phi_{C}(X), A^{c^1}(C,l) and P^{c^1}(C,l) are unclear to me, and seem to be inconsistent from one proof to another. Partly because the definition of C at different stages of the algorithm can change. A^{c^1}(C,l) and P^{c^1}(C,l) became more unclear to me when the authors refer the readers to lines 6-11 of the algorithm; for a moment, I thought "l" denotes the length of the markov chain at an iteration. Maybe the authors can remove this reference. Also, I'd be great if the authors can provide more intuition of their analysis, on a high level.


Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences
We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with M ≥ 3 components. Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical Gaussians. We prove that the log-likelihood value of these bad local maxima can be arbitrarily worse than that of any global optimum, thereby resolving an open question of Srebro [2007]. Our second main result shows that the EM algorithm (or a first-order variant of it) with random initialization will converge to bad critical points with probability at least 1 − e −Ω(M ) . We further establish that a first-order variant of EM will not converge to strict saddle points almost surely, indicating that the poor performance of the first-order method can be attributed to the existence of bad local maxima rather than bad saddle points. Overall, our results highlight the necessity of careful initialization when using the EM algorithm in practice, even when applied in highly favorable settings.

The paper is concerned with the existence of local modes in the log-likelihood of Gaussian mixture models. The simplest possible mixture model is considered: isotropic Gaussian components with known scale. In addition the model is well-specified. The authors show that local modes exist, even when infinite data is available, and that these are arbitrarily bad, in term of differences in log-likelihood between the local and global optima. The paper focuses on particular configurations, in particular cases where the mixture centres are divided between well-spaced clusters. Under such configurations, the authors show that random initialization will make so that EM misses the global optima almost certainly, as the number of components increases.
The paper is well written, with occasional typos. I have two main doubts about the practical relevance of the paper.  1. Theorem 1 shows that the log-likelihood has a local modes, even when infinite data is available. The authors show that this mode exists when a particular configuration of the centres is chosen. My question is: how likely is this configuration to occur in practice? I would feel more confident about the practical relevance of the paper if, for instance, the authors made some assumptions about the distribution of the centres, and then showed that "bad" configurations become increasingly likely as the number of components (k) or the dimensionality of the problem increases. L210 to L218 of the paper say that Theorem 2 addresses these concerns, but I am not sure it does. It would be good to have a clear statement say how likely "bad" configurations are, under some assumptions. Notice that here I am talking about "bad" configurations (positions of the true centres), not "bad" initializations.  2. In theorem 2 the initialization of the EM algorithm comes to play.     The authors consider initializing the centres of the components by randomly sampling the mixture. With this initialization, and under specific configurations of the centres, the EM very often fails to converge to the global optimum. Now, let us consider the case where the data is generated using 3 Gaussian, 2 of which are close and one far apart. If we have the same number N of data-points from each Gaussian, then 2N points will fall close to the first cluster and N will fall close to the 3rd density. It seems to me that very odd that somebody could initialize EM by putting 2 centers where there are only N points and 1 center where there are N observations, especially given that the two clusters are very well separated from each other. Is random initialization practically relevant when the data is clearly clustered?   L74-81 I am not sure what the example on lines 74-81 is meant to convey. It does not convince me that Sresbro conjecture holds when k = 2, so what is its purpose?  L233-235 I understand requirement (2) here, but I don't see why requirement (1) is needed. You mean that, if you initialize a center between 2 clusters, you cannot guarantee that it will move toward the correct cluster?  L267-273 The authors prove that EM is unlikely to converge to a saddlepoint. From a practical point of view, why is this result important? Is converging to a saddlepoint more or less problematic than converging to a local optima?  MINOR POINTS:  L18 "We further show gradient EM algorithm" add "that the" before "gradient".  L38 "quality of the estimate" maybe "estimates" is better.  L60 "to characterize when it is that efficient algorithms..." I guess here you really mean "to characterize under which conditions efficient algorithms..."  L138 Just to be clear: O(d^1/2)-separated mean that all densities are c-separated, with c being O(d^1/2).  L172 Isn't this sentence redundant? Please reformulate it or remove it.  L177 "this is equivalent of sampling " maybe "equivalent to" is better.  L218 I don't understand what a "well-separated" constant is.  L220 " initialized by the the random initialization"  L225 I don't know what \Omega (k) means.  L245 "unfortunate coincident for one single algorithm" used "coincidence" instead.  L249 "Recall for uniform weighted" add "that" after "recall"  L308 "And claim when" add "that" after "claim"


Causal Bandits: Learning Good Interventions via Causal Inference
We study the problem of using causal models to improve the rate at which good interventions can be learned online in a stochastic environment. Our formalism combines multi-arm bandits and causal inference to model a novel type of bandit feedback that is not exploited by existing approaches. We propose a new algorithm that exploits the causal feedback and prove a bound on its simple regret that is strictly better (in all quantities) than algorithms that do not use the additional causal information.

I have read the author feedback, and stand by my recommendation for accepting the submission as a poster. The paper studies a novel problem formulation that can apply to several real world problems (e.g., many applications that are currently modeled as contextual bandits). After reading the author feedback, I am suitably convinced that this formulation is not trivially solved by partial monitoring or Bareinboim et al, 2015 etc.  I can see several follow-ups that study the connections between graph properties of the underlying causal graph and simple/cumulative regret and jointly learning causal structure online, etc. more thoroughly. Alg2 and its assumptions are unfortunately too limiting, but the basic ideas can spur further research.   --Earlier review follows below--  The paper studies best arm identification in multi-armed bandits, where on each round, a reward as well as a bunch of covariates are revealed to the learner (like in partial monitoring games). These covariates influence the reward through a known causal model, and this additional information is used to develop an algorithm with minimax optimal simple regret (up to log factors) for "parallel" causal graphs (covariates independently influence reward). Moreover, an algorithm that requires additional information about conditional intervention distributions (to estimate a quantity m < #actions) provably achieves \tilde{O}(sqrt(m/T)) simple regret. Simulations show that these methods can indeed perform better than the Successive Reject algorithm by knowing the correct causal model.
I have read the author feedback, and stand by my recommendation for accepting the submission as a poster. The paper studies a novel problem formulation that can apply to several real world problems (e.g., many applications that are currently modeled as contextual bandits). After reading the author feedback, I am suitably convinced that this formulation is not trivially solved by partial monitoring or Bareinboim et al, 2015 etc.  I can see several follow-ups that study the connections between graph properties of the underlying causal graph and simple/cumulative regret and jointly learning causal structure online, etc. more thoroughly. Alg2 and its assumptions are unfortunately too limiting, but the basic ideas can spur further research.   --Earlier review follows below--  The paper was clear to follow and introduces a novel model in multi-armed bandit problems with well-analyzed algorithms. I appreciate that code implementing the experiment was available in the supplementary. My comments below focus only on the weaknesses.  The paper motivates a nice model of reward-feedback in the multi-armed bandit problem. However, please elaborate on the differences between this model and combinatorial partial monitoring (e.g. Tian Lin et al, Combinatorial Partial Monitoring Game with Linear Feedback and Its Applications, ICML 2014). In particular, are there suitable choices of encoding the outcome vector, action matrices etc. in CombPM that recovers every causal model instance? Or the other way around? Or are the two incomparable (with a counter-example in each direction)?  The simulated experiment was a weak point of the paper. First I don't understand why lil'UCB was not simulated (rather than/in addition to Successive Reject). Secondly, I do not understand the comment about Alon et al's algorithms for adversarial bandits with feedback graphs. Yes a malicious adversary can set q = 0 but I don't see why Algorithm 1 or 2 (or their variant) do any better than their algorithm in that case (m(q)=N then). Thirdly, the terrible performance of Alg 1 and 2 in Fig 2(a) only convinces me that the asymptotic regret analysis is giving us an incomplete picture of these algorithms' performance (for instance, being reward oblivious is nearly minimax optimal simple regret!) I would like to see a discussion on how better to characterize their performance (e.g. would cumulative regret be the right thing to tell these algorithms apart? Would the idea sketched in Section 6 then be nearly minimax-optimal too)?  Assuming that conditional intervention distributions is known is, as the paper admits, quite limiting. I can also intuitively see that we need some way to link the action we take to how it affects the outcomes for the parents of Y in the causal graph (as also the reward outcome Y). Is this assumption equivalent to saying, the results provably work only for causal models where all non-Y nodes are parents of Y (so, a parallel graph with additional edges between X_i) and intervening on some subset of the non-Y nodes are the allowed actions? If so, I find the chain graph example distracting (yes, multiple intervention rewards can be observed there, but we cannot really exploit that without additional knowledge).  Interesting choice of truncation, and I found the comment about empirical results when B_a is set higher intriguing. Am I correctly understanding that clipping as Y_t min{B_a, R_a(X_t)} rather than Y_t R_a(X_t) I{R_a(X_t) < B_a} will introduce less bias and could work better (although worst-case analysis does not change)? On line 280: no truncation was performed for Algorithm 2, so could it also be unbounded variance hurting Alg 2?  Minor comments:  201: P{X_1 = 1} = 0. 321: "reach round" -> each


Bayesian latent structure discovery from multi-neuron recordings
Neural circuits contain heterogeneous groups of neurons that differ in type, location, connectivity, and basic response properties. However, traditional methods for dimensionality reduction and clustering are ill-suited to recovering the structure underlying the organization of neural circuits. In particular, they do not take advantage of the rich temporal dependencies in multi-neuron recordings and fail to account for the noise in neural spike trains. Here we describe new tools for inferring latent structure from simultaneously recorded spike train data using a hierarchical extension of a multi-neuron point process model commonly known as the generalized linear model (GLM). Our approach combines the GLM with flexible graph-theoretic priors governing the relationship between latent features and neural connectivity patterns. Fully Bayesian inference via Pólya-gamma augmentation of the resulting model allows us to classify neurons and infer latent dimensions of circuit organization from correlated spike trains. We demonstrate the effectiveness of our method with applications to synthetic data and multi-neuron recordings in primate retina, revealing latent patterns of neural types and locations from spike trains alone.

The authors describe a statistical model of neural data that combines random network models (distributions over the connections and weights between neurons) with a standard point process spiking model, a generalized linear model (GLM). By restricting the spiking process to logistic spike count models, the authors apply the Polya-gamma augmentation trick to perform efficient inference through Gibbs sampling. The authors apply their technique to simulated data (recovering latent network structure in the simulated network) and to spiking activity of retinal ganglion cells (RGCs). This technique lets the authors quantitatively reason over network connectivity, providing a way to reason about hypotheses regarding how network structure relates to recorded neural activity.
In general, the motivation, model, and experiments are clearly described and well presented.  The usefulness of the application to retinal data is overstated. In the second paragraph, the authors mention that there are a "plethora of RGC types" that have been identified and then go on to discuss that their method will "automatically discover this structure from correlated patterns of neural activity". This is incorrect for two reasons: First, the authors demonstrate that their method only discovers two RGC types (the standard ON vs OFF classification) and their results do not recover the "plethora of RGC types" mentioned in the introduction. Secondly, at least in the retina, the network activity is directly tied to the stimulus--and the experimental choice of stimulus--and therefore their method is, implicitly, tied to the choice of stimulus. For example, consider if the experiment had used a full field stimulus or only concentrated the stimulus on a small part of the visual field. Then, the network activity (and resulting conclusions of the modeling effort) would be markedly different. I am just pointing out that when analyzing responses of sensory areas the analysis is not independent of the stimulus, and the authors should discuss and highlight the implications of the experimental stimulus.  Another point about the application to retinal data--the authors make a point of highlighting that their methods discover ON and OFF cell types in the data without knowledge of the stimulus using a minute of data (first paragraph of section 5). However, experimentally, we *do* have access to the stimulus (in fact, we choose it) and it does not make sense to perform stimulus independent analysis (especially for something like the discovery of OFF or ON cell types, which can be done in seconds using a light and dark flash stimulus).  However, I do recognize that the stimulus independent analysis is very useful in two very important contexts: (1) deeper in the brain, where the relevant stimulus may be unknown or hard to control, and (2) in sensory recordings in response to uncontrolled stimuli, such as natural stimuli. These are both scenarios where the techniques put forth in this paper could prove to be powerful ways of identifying latent structure that has been difficult to crack using more conventional stimulus-response modeling approaches. I think making this point explicit in the introduction or conclusion would further strengthen the motivations of the paper.  As a final point on the comparison to retinal data, I would appreciate if the authors compared their technique to some naive baselines. For example, the correlation between the spike trains of two cells drops off with distance in the retina. If one simply used the spike train correlation as an estimate of distance, how would that compare to the NGLM approach in Figure 4b? In addition, how does the network structure compare to a simpler approach of the MAP estimate of a GLM with l1 or l2 priors on the coupling terms? The authors mention this simple model at the end of section 2.2, it would be nice to know how inferred network structure from a regularized GLM compares to their NGLM technique.  I would have appreciated more discussion surrounding the types of network models used (e.g. the stochastic block model and latent distance model). These were introduced very briefly in section 2.3, and I would have appreciated more explanation of what kinds of latent structure these descriptions could capture and how they relate to existing hypotheses of connectivity in neural systems.  In general, the paper is well written, I enjoyed reading it, and it raises interesting questions about how to model network structure given recorded spiking activity. To really demonstrate the advantages of their approach, more comparisons to simpler and existing approaches are needed. More discussion about how their methods will be relevant for other neural systems besides the retina is also welcome.


A Probabilistic Programming Approach To Probabilistic Data Analysis
Probabilistic techniques are central to data analysis, but different approaches can be challenging to apply, combine, and compare. This paper introduces composable generative population models (CGPMs), a computational abstraction that extends directed graphical models and can be used to describe and compose a broad class of probabilistic data analysis techniques. Examples include discriminative machine learning, hierarchical Bayesian models, multivariate kernel methods, clustering algorithms, and arbitrary probabilistic programs. We demonstrate the integration of CGPMs into BayesDB, a probabilistic programming platform that can express data analysis tasks using a modeling definition language and structured query language. The practical value is illustrated in two ways. First, the paper describes an analysis on a database of Earth satellites, which identifies records that probably violate Kepler's Third Law by composing causal probabilistic programs with nonparametric Bayes in 50 lines of probabilistic code. Second, it reports the lines of code and accuracy of CGPMs compared with baseline solutions from standard machine learning libraries.

The authors introduce conditional generative probabilistic models (CGPMs), which are essentially probabilistic models with a partially specified conditional dependence structure. A CPGM can embed submodels for which the conditional dependence structure need not be known, as along as it is possible to sample and evaluate the density for any conditional distribution on a subset of leaf node variables in the model. CGPMs can be composed into larger CGPMs, for which approximate implementations for the sampling and density functions can be defined using an importance sampling method. The authors present case studies of analysis performed using CGPMs that are implemented with BayesDB and Venturescript.
I think this is a strong paper. The authors on the one hand define a clear mathematical abstraction, the CPGM, and on the other hand show case studies of analysis performed with a probabilistic programming system that implements this abstraction. The CPGM is described in reasonably precise mathematical notation, which formalizes the aspects of this work that generalize beyond the specific implementation. The cases studies are not easy to follow without familiarity with the BayesDB system, but do a good job of illustrating the general idea to this approach, which is to provide a high level modeling/inference abstraction in which black box components can be embedded. The paper is reasonably clear. It is fairly heavy on notation, but most aspects of the presentation can be understood on a sufficiently close reading. I have pointed out point where presentation can be improved below.   Questions and Suggestions  - It would be helpful to explicitly define what is meant by "query" and "evidence" in the context of this paper, which currently has to be inferred by the reader from the simulate / logpdf interface.   - What is the distinction between the observed output nodes D and the evidence node x_{r,E}? Does D refer values {x_r: r \in D} for which *all* variables are *always* observed?  - In figure 1, why is there a plate around x and not around z? I would avoid using the the index i when writing {z_i : i \not= r} in the caption, since the authors use x_[r,i] in the next sentence.  - I was unable to find a definition for {q_j} in the simulate interface. Presumably q_j \in [T]? Likewise the notation x_[r,Q] presumably refers to {x_[r,q] : q \in Q}?   - What is meant by "the subset of the outputs x^a_[*, A]"? In particular, what does the "*" notation mean?  - In algorithm 1, please define explicitly which symbol is bound to which input in the functions simulate and logpdf. Having to infer this from context makes it not obvious what is going on.   - I'm confused by the simulate implementation. Line 3 generates a sample conditioned on both the evidence and the queried variables, does it not? So then, why would {x_[r,Q^k] \in s^m}?, since in line 24 we simulate \E^k? Should line 3 perhaps read      Weighted-Sample(x[r,E_k])  - This is a matter of taste, but I would prefer for the authors to not use 'w' to describe a log weight, since most of the importance sampling literature uses 'w' to refer to an importance weight (i.e. the exponent of the log weight w used by the authors).   - evdience -> evidence (please run a spell check)


Understanding Probabilistic Sparse Gaussian Process Approximations Mark van der Wilk †
Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.

The paper gives theoretical analysis of FITC and FVE approximations for sparse large-scale Gaussian processes. There are various analytical results and proofs that are interesting.
The paper analyses the differences between the two main large-scale approximations strategies for GP inference, namely the variational inference of Titsias'09, and the FITC by Snelson'06.  It's an interesting paper that tries to summarise how these two main approaches differ and how they affect the resulting models. The paper is limited in that it only covers how these two approximation methods affect the resulting models, but does not comment on the running times at all. The only motivation for sparse approximations is large-scale problems, and thus scalability. Perhaps for this reason they do not consider e.g. EP and SVI approaches, which have their own interesting approximation effects the resulting model as well. A longer format might fit this paper better, with analyses of all currently used approaches instead of just these two.  All the analyses are interesting, especially regarding the local minima, heteroscedastic noises and addition of inducing points. 3.4. illustrates beautifully the behaviors of FITC/FVE. It seems that FVE is superior, but then in 3.5. a real dataset shows FITC matching or surpassing VFE performance. To explain this discrepancy more large-scale datasets should have been analysed to see if FITC consistently beats VFE, and in which settings. The theoretical drawbacks of FITC would be cast in strange light if it does not seem to suffer from them in practise, as hinted in table 2.  I'm also having problems with the running example regarding FITC. Fig1 shows the FITC to give a badly "pinched" model. The same dataset was used in many other papers (with roughly comparable experimental settings), where this does not happen for FITC. In both Snelson'06 and Titsias'09 the FITC gives a good non-pinched model for this data, and in Quinonero'05 the FITC gives slighly pinched model, but not as bad as in the present paper. This is likely due to number of datapoints (fig3 again is not pinched), so it would be nice to see how the "pinching" effects behaves as a function of data size.  Overall it's an excellent paper with good findings on FITC/FVE, but the theory and practise seem to disagree in large-scale cases, and there is no running time analysis. The paper would also be greatly strenghtened by being more comprehensive in compared methods (eg. EP/SVI).


Visual Question Answering with Question Representation Update (QRU)
Our method aims at reasoning over natural language questions and visual images. Given a natural language question about an image, our model updates the question representation iteratively by selecting image regions relevant to the query and learns to give the correct answer. Our model contains several reasoning layers, exploiting complex visual relations in the visual question answering (VQA) task. The proposed network is end-to-end trainable through back-propagation, where its weights are initialized using pre-trained convolutional neural network (CNN) and gated recurrent unit (GRU). Our method is evaluated on challenging datasets of COCO-QA [19] and VQA [2] and yields state-of-the-art performance.

This paper studies the task of visual question answering. In order to answer questions about the images, this work derives their model from neural reasoners. Neural reasoners can take a question, and supporting facts necessary to answer the question and produce updated query based on the refinement provided by the supporting facts. This task proposes to use object detection (with their spatial position) as facts necessary to answer the question. The results on standard test datasets for VQA show some promise but do not match current state of the art methods performance.
The paper is well written and the experiments appear sound. The overall motivation of having more directed and detailed question rather than a vague one is also important for VQA. The method is derived from neural reasoning system previously developed for text based QA. The authors do not offer any improvements or modifications to the reasoning system but rather use objects in the image as facts instead of textual facts in traditional neural reasoning system. Hence, the overall novelty of the proposed method is minor.   Additionally, I am not fully convinced that object detection boxes alone provide enough facts necessary for answering the questions in the current datasets. Firstly, such method invariably perform poorly for counting questions which the authors also make note of (line 220). Second, the object and their locations should only be helpful for questions directly related to objects. The authors contend that 'most of the questions are related to objects' (line 97-99), but that is not the case for the VQA dataset. Many questions refer to object but are actually seeking information near the object (What is next to the couch? What is to the left of the dog? etc..). Moreover, the object proposals are not drawn based on the entities referred in the question but rather by using automated 'top-ranked' regions. So, the use of objects in the scene (even when used in conjunction with their location) serves only as limited source of knowledge. Since the entire set of 'facts' also include the whole image, it is not possible to study the effects of including the edge boxes. The authors do not present ablations studies to prove that the inclusion of facts do indeed help the reasoning system. It could be done by training a controlled system devoid of 'overall' image feature and only consisting of the object proposals.   On the same token, the attention map is also produced on the top-ranked image regions that are generated without knowledge of the question. A grid-layout (often obtained by CNN networks before pooling layers [1] [2] etc..) is also not used which means that certain image regions (that are not bounded by the top-ranked regions) are unavailable for attending. The resolution provided by including the whole image does not provide granularity required for attending to specific parts.  Similarly, the authors show a number of examples of question representation update where the updated question, at times, show greater degree of detail in the question. However, I am not convinced that this can be taken as indication of successful representation update for several reasons. 1) The authors use nearest neighbor approach to find the closest question in the dataset that match the updated question representation. It is unsurprising that the updated representations contain references to objects present in the image since the input to the reasoning system consist of explicit object locations and features from them. It is, however, uncertain whether the updated representation actually asks about the same entity as the original question. Certainly, the updated question contain more detail but is the meaning of the question unchanged to the original? 2) What do the failure cases look like? 3) The examples only include questions from COCO-QA dataset, which are derived from captions via NLP algorithms and are notoriously bad in terms of grammar, composition and phrasing.   The above comment does not, however, mean that the updated representation wont produce better query. This only means that the proposed nearest neighbor method to assess or demonstrate method's efficacy is dubious.  In a nutshell, the paper is well-written and presents a interesting method of using reasoning engine for VQA. However, the authors do not offer any new innovation on the neural reasoning and the image based fact generation proposed by the authors may be inadequate to completely reason about the images. The general applicability of the work in current form is low, especially considering similar methods offer better performance in VQA. For a stronger case, the visual facts need to be question specific and more detailed so that it can offer better basis for reasoning. The explicit efficacy of the proposed work also should be demonstrated by the help of ablation studies.   ----------- References ----------- 1.Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, 2016.  2.Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.


On Graph Reconstruction via Empirical Risk Minimization: Fast Learning Rates and Scalability
The problem of predicting connections between a set of data points finds many applications, in systems biology and social network analysis among others. This paper focuses on the graph reconstruction problem, where the prediction rule is obtained by minimizing the average error over all n(n − 1)/2 possible pairs of the n nodes of a training graph. Our first contribution is to derive learning rates of order O P (log n/n) for this problem, significantly improving upon the slow rates of order O P (1/ √ n) established in the seminal work of Biau and Bleakley (2006). Strikingly, these fast rates are universal, in contrast to similar results known for other statistical learning problems (e.g., classification, density level set estimation, ranking, clustering) which require strong assumptions on the distribution of the data. Motivated by applications to large graphs, our second contribution deals with the computational complexity of graph reconstruction. Specifically, we investigate to which extent the learning rates can be preserved when replacing the empirical reconstruction risk by a computationally cheaper Monte-Carlo version, obtained by sampling with replacement B n 2 pairs of nodes. Finally, we illustrate our theoretical results by numerical experiments on synthetic and real graphs.

This paper considers a generative model of a random graph. Graphs are generated as follows. We start with a set of vertices. For each vertex, we independently and identically sample a feature from a vector space according to a law \mu. A stochastic kernel, \eta, determines the Bernoulli probability of an edge being present, conditioned on two (unordered) vertex features. For each pair of vertices, we use \eta to sample the Bernoulli variable thus determining whether there is an edge or not.  We have no a priori knowledge of \mu or \eta, except for a fully observed (features+edges) training graph with n vertices. In a partially observed (features only) test graph, we would like to infer where there are edges. We focus on a simplified test scenario, where two features (X1,X2) are sampled independently according to \mu, and an edge E is determined according to \eta(.|X1,X2). The learning goal is as follows: give the n-vertex training graph, construct a binary function g_n that maps two features into a label (edge present or absent). The performance of g is determined by the probability of error: R(g_n)=\Prob\{ g_n(X1,X2) \neq E \}.  If \eta is known, the Bayes (best) risk-achieving g^* can be obtained. To avoid overfitting, one would generally restrict the model of possible gs (say to VC dimension V), and then compare to the benchmark of the best-in-model, g'. The paper shows that g_n obtained by empirical risk minimization is only O(log(n)/n) off from g', in the following sense: there is a constant b, such that for all a > 1, w.p. > 1-d:  R(g_n)-R(g^*) \leq a (R(g')-R(g^*)) + (a+1/(a-1)) b V log(n/d)/n  This establishes an effective convergence rate of O(log(n)/n), improving on the previously best known rate of O(1/sqrt(n)).  The authors also propose a sub-sampling based method to scale the approach and give some numerical experiments to support the theory.
- The paper is generally very well written. The result is interesting and solid, and potentially impactful. It is however a bit strange that the authors sell their results from the perspective of "fast rates despite no margin conditions" (presumably because of the proof techniques), while the crux of it is that "dependent samples are almost independent". This is because the number of (albeit not independent) samples is not n, but rather N=n(n-1)/2, and therefore we have a "slow" rate for N.  - When proposing the edge-sampling scaling technique, the authors claim that "sampling vertices directly would greatly damage the learning rate". The evidence that they give toward this is a variance comparison, on Line 63 in the supplementary material. First, the "is of order 1/m" claim on that line is not obvious (and probably incorrect). Second, if we forget sub-sampling and use (the even simpler) truncation method by discarding all but the first m vertices and the resulting sub-tree, then Theorem 1 applies as is, and we get an end-to-end rate of log(m)/m. This would correspond to B=m(m-1)/2 pairs, and thus we get the same rate of 1/sqrt(B) of Theorem 5 (without even the intermediary of the larger sample size n). Additionally, Table 1 (in both the main text and supplements) and Figure 1 (in the supplements) do not appear to show a difference in rate. Thus it would seem that the main "damage" is to the constants, not to the rate as claimed. The only other claim in this regard (Lines 244-246 in the supplementary material) is that the gap widens with increased model complexity. This wider gap could simply be due to the fact that both excess risks get multiplied by the VC dimension (so it could be a wider additive gap, while the multiplicative gap in excess risk remains constant).  Minor remarks:  - In Line 92, I think what is meant is "Marginalized over the features" and not "Conditioned upon the features". Once the features are specified, the edges are independently generated, though not identically. This should be stated plainly. On the other hand, of course the data point triplets Z_{ij}=(X_i,X_j,e_{ij}) are not independent (though marginally identically distributed).  - The term on Line 169, Lambda(g)=R(g)-R^*, may understandably be called excess risk. However, that terminology is often reserved for the excess over the best-in-model, rather than the Bayes risk. Especially given that the latter terminology is also used in the paper (Line 122), perhaps a different choice of words should be used on Line 169.


DECOrrelated feature space partitioning for distributed sparse regression
Fitting statistical models is computationally challenging when the sample size or the dimension of the dataset is huge. An attractive approach for down-scaling the problem size is to first partition the dataset into subsets and then fit using distributed algorithms. The dataset can be partitioned either horizontally (in the sample space) or vertically (in the feature space). While the majority of the literature focuses on sample space partitioning, feature space partitioning is more effective when p n. Existing methods for partitioning features, however, are either vulnerable to high correlations or inefficient in reducing the model dimension. In this paper, we solve these problems through a new embarrassingly parallel framework named DECO for distributed variable selection and parameter estimation. In DECO, variables are first partitioned and allocated to m distributed workers. The decorrelated subset data within each worker are then fitted via any algorithm designed for high-dimensional problems. We show that by incorporating the decorrelation step, DECO can achieve consistent variable selection and parameter estimation on each subset with (almost) no assumptions. In addition, the convergence rate is nearly minimax optimal for both sparse and weakly sparse models and does NOT depend on the partition number m. Extensive numerical experiments are provided to illustrate the performance of the new framework.

The paper presents a feature-wise partitioning approach for distributed sparse regression. Unfortunately, the results are rather incremental for the level of NIPS, as the result only holds for random design matrices, and the paper in its current form lacks discussion of several lines of related work and experimental baselines.
The paper presents a feature-wise partitioning approach for distributed sparse regression. Unfortunately, the results are rather incremental for the level of NIPS, as the result only holds for random design matrices, and the paper in its current form lacks discussion of several lines of related work and experimental baselines.  While I do definitely like the conceptual idea of the partitioning followed by de-correlation, the presented theory falls short of expectations as it only holds for random design matrices.  The paper however does not clearly explain the novelty and differences over [9]. Also, in addition to [9], relations to related work [B,C] are not sufficiently discussed in the current version. Almost matching upper and lower bounds for the amount of communication are in the meantime well-studied [C]. The approach presented here needs to be set into this relevant context as well.  Clarity of presentation can be improved.  The presented experiments are comprehensive in terms of datasets used, however not comprehensive in term of competing algorithms (many distributed solvers for L1 exist and the most prominent ones should be included in the comparison).    MINOR: - abstract, line 6 "feature space partitioning is more effective when p >> n". This should be supported by a result. Might need to tune down, as most algorithms scale with p*n no matter which of the two possible partitionings is chosen. - conditional number -> condition number      REFERENCES:   [A] Arjevani, Y., & Shamir, O. (2015). Communication Complexity of Distributed Convex Learning and Optimization (pp. 1747–1755). NIPS 2015 - Advances in Neural Information Processing Systems 28.  [B] Heinze, C., McWilliams, B., & Meinshausen, N. (2016). DUAL-LOCO: Distributing Statistical Estimation Using Random Projections (pp. 875–883). Presented at the AISTATS - Proceedings of the th International Conference on Artificial Intelligence and Statistics.  [C] McWilliams, B., Heinze, C., Meinshausen, N., & Krummenacher, G. (2014). LOCO: Distributing Ridge Regression with Random Projections. arXiv   UPDATE AFTER REBUTTAL: I thank the authors for clarifying the relation of the idea to [B,C]. I believe the authors can strengthen the paper for a future submission along the lines suggested.


Improved Regret Bounds for Oracle-Based Adversarial Contextual Bandits
We propose a new oracle-based algorithm, BISTRO+, for the adversarial contextual bandit problem, where either contexts are drawn i.i.d. or the sequence of contexts is known a priori, but where the losses are picked adversarially. Our algorithm is computationally efficient, assuming access to an offline optimization oracle, and enjoys a regret of order O((KT )

The authors studies the contextual MAB problem. The features across the periods are generated i.i.d. from a distribution known to the decision maker, but the losses across the periods are adversarially generated. By designing a new "partial information relaxation" and using a recent framework for contextual MAB proposed by Rakhlin and Sridharan, the authors manage to improve the regret from the state of the art, namely O(T^(3/4)), to O(T^(2/3)).   The key to improvement is the incorporation of a Rademacher term (the middle term) in the partial information relaxation defined between Line 129 and 130. This term has two Rademacher random variable, Z_t and epsilon_t. My understanding is that the rv Z_t ensures that the partial information relaxation is valid (satisfying eqns (2, 3)), while the rv epsilon_t ensures that the relaxation term grows in a rate of sqrt(T) (conf Lemma 2) instead of T, which brings about the improvement.
The paper makes some interesting contribution by proposal a new partial information relaxation to improve the regret bound. The analysis of the partial info relaxation bears some similarity to that in Rahklin and Sridharan, but the inclusion of the Radamacher term is new. By using Lemma 2, the bound is improved. While this is definitely a new result and there are new ideas, the similarity to current literature kind of compels me to give a '3' instead of a '4' for the Novelty/Originality and Technical Constribution scores.  Presentation: The paper is concise and well written, but I think that the proof of admissability between 5 and 7 is too "straight-lined". The unraveling  and bounding from Line 156 to Line 188 is rather long, but it would be more readable if the authors first highlight the major steps in the calculation, and then explain how to go from one major step to the next. This will give a better feel on your design of partial information relaxation. In particular, I think you should label "the quantity inside the expectation" between Line 162 and 163 to avoid ambiguity. A clear presentation of the proof is very important, since it will guide future readers on the design of partial information relaxation in the framework.


End-to-End Kernel Learning with Supervised Convolutional Kernel Networks
In this paper, we introduce a new image representation based on a multilayer kernel machine. Unlike traditional kernel methods where data representation is decoupled from the prediction task, we learn how to shape the kernel with supervision. We proceed by first proposing improvements of the recently-introduced convolutional kernel networks (CKNs) in the context of unsupervised learning; then, we derive backpropagation rules to take advantage of labeled training data. The resulting model is a new type of convolutional neural network, where optimizing the filters at each layer is equivalent to learning a linear subspace in a reproducing kernel Hilbert space (RKHS). We show that our method achieves reasonably competitive performance for image classification on some standard "deep learning" datasets such as CIFAR-10 and SVHN, and also for image super-resolution, demonstrating the applicability of our approach to a large variety of image-related tasks.

This paper provides a supervised manner to leran a multilayer kernel machine, which is an extension of convolutional kernel networks (without supervision). Its major purpose is to couple learning and data representation together within the context of kernel machines.
This paper proposes an original idea and theoretically appealing solutions to solve it.   Quality:  Its first part (Section 1, 2) is excellent, but its latter part may be a little weak.   Section 3 is a little dense with numerous details and heuristics. A pseudo-code showing the overall framework may be helpful for readers.  Section 4 is a little short to validate the potential effectiveness of the proposed method. A direct comparison between CKN and CNN with same architecture may be more informative for readers.  Originality:  The purpose of this paper is appealing and its solutions are novel. It is worth an effort to explore the bridge between kernel machines and deep neural networks. But experimental results may not be enough to express its effectiveness fairly. Several direct comparisons will make this paper a more influential one.  Potential impact:  Its motivation is important, but more well-designed experiments will definitely make the effectiveness of this paper easier to accept.  Clarity:  I think Section 3 a little dense, a pseudo-code summarizing details will be welcome.  Detailed Comments:  (1) In the context of supervised learning, this paper seems to put less efforts to explain why the proposed solution would be superior to a standard CNN. I am not sure its natural regularization function (the norm in the RKHS) will be the key to make itself a competitive alternative to CNN.  (2) Since the superiority may not be easily evaluated in an analytical form, it will be more informative to validate them empirically. Otherwise, the discussion may, by chance, mislead readers falling into the assumption that a CKN is doomed to be better than a CNN.  (3) The discussion in the image classification experiments (Section 4.1) may not be sufficient. For example, why a 9-layer network is chosen? Will a shallower CKN be more suitable for CIFAR-10?  Moreover, it will be more informative to provide the performance of CNNs with same architecture. It can be understood if dropout and batch-normalization are removed in the training course of CNN, but it is important to show the superiority of supervised CKN in the same architecture.  Also, it will be beneficial to show the performance gain compared to its unsupervised counterpart (learnt by a k-means algorithm in all layers). Besides, the k-means algorithm is sensitive to initialization, whether the supervised CKN will still achieve competitive performance when the result of k-means is not good. Repeating the experiment several times or tuning the network from a random initialization will give more insights to readers.  At last, I have several personal curiosities about the method. a) How many patches is required for the k-means algorithm to output a good initialization? b) Whether transferring weights from a pre-trained network will boost its performance. c) Whether fine-tuning the 2-layer CKN-GM or CKN-PM in [18] will show improvements. I personally think this is the key to show the effectiveness of the proposed solution.  (4) The image super-resolution experiment may have the same problem. Why is a 9-layer network preferred? Since [8] claimed that a deeper network is hard to output a better result compared to a shallower one, it will be beneficial to discuss the reason that the supervised CKN can tackle this problem. I also wonder whether the performance of a 3-layer CKN will outperform its counterpart CNN in [8].  Moreover, in the context of super-resolution, inference time is also important to evaluate a method. I think the 9-layer CKN with 128 filters for each layer may be too computationally heavy for this task. It is helpful to provide a visualization of the trade-off between accuracy and inference time for different methods like Figure 1 in [26].  Summary:  Overall, it is a promising paper with potentially huge influence. Due to the limited space, a tradeoff between details and experiments will surely make itself a shining one in the NIPS.


InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that also maximizes the mutual information between a small subset of the latent variables and the observation. We derive a lower bound of the mutual information objective that can be optimized efficiently. Specifically, InfoGAN successfully disentangles writing styles from digit shapes on the MNIST dataset, pose from lighting of 3D rendered images, and background digits from the central digit on the SVHN dataset. It also discovers visual concepts that include hair styles, presence/absence of eyeglasses, and emotions on the CelebA face dataset. Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing supervised methods. For an up-to-date version of this paper, please see https://arxiv.org/abs/1606.03657.

This paper presents an extension of GAN to learn interpretable representations (codes to generate images). The main observation is that the original GAN tends to ignore the additional condition (code) to generate images P(x|z) = P(x|z,c) where c is the code. To address this issue, this paper proposes to maximize the mutual information between the code c and the generated images G(z,c), and derive a variational lower bound for learning with a proposal posterior distribution Q(c|x). The final learning objective is a combination of minmax game loss of GAN and log likelihood of the proposal code distribution. In network implementation, this new code likelihood term is simply added at the end of discriminator with additional linear layers. Experiments are carried out on MNIST, renderings of faces and chairs, SVHN and CelebA with codes being categorical or uniform.
1. ----------------Technical quality: -The descriptions from Line 64 to 77 seem a bit confusing. What I understand is [22] and [23] are weakly-supervised methods while [24] and [7] are actually fully-supervised methods. -The comment in Line 79 seems to be inaccurate. Another notable unsupervised method for disentangling is the work by Cheung et al.: Discovering Hidden Factors of Variation in Deep Networks, ICLR 2015.  -In Line 165, this paper claimed to use factorized Gaussian as code distribution, but it turned out that it used uniform distributions through all the experiments. This should be fixed. -One question about the formulation is about z, the free random vector. I wonder whether z could be considered as the same role as c so that the model only needs to model the mutual information between z and x. If not, what role is z playing in the G(z,c) model? -The proposed algorithm seems to achieve nice results on MNIST: different numbers are well separated in the categorical code. Interesting, the paper actually provides the knowledge about total number of categories for the code distribution (K). This information could be privileged in real-world. I wonder what results can be obtained if the K value is altered.  -This paper presents many qualitative results on five datasets but little quantitative results (only Figure 1 and classification error on MNIST in Line 204). Quantitative evaluation of generated samples has been challenging, but considering the main argument of this paper is to learn disentangled representations for downstream applications, some classification experiments on faces, chairs and house numbers using learned features will be really valuable. 2. ----------------Novelty: +The information theory based derivation of learning disentangled codes is somewhat novel, although it bears similar theoretical background with wake-sleep algorithm and variational auto-encoders (VAEs) regarding joint learning of inference and generation networks. In VAEs, the sampling chain is image -> code -> image while the sampling chain used in this paper is code -> image -> code. Having code in the middle restricts the form of code distribution for enabling end-to-end training. However, in this paper having the code at the two ends of the network grants a flexible use of different code distributions such as categorical or uniform distributions. I believe this is the main advantage of the proposed algorithm. 3. ----------------Impact: +Learning interpretable and disentangled representations with generative models is an important research issue. The maximization of mutual information between code and image provides new insights and improves a popular generative model (GAN). I believe this has certain impact in academics.  4.-----------------Presentation:


Feature selection in functional data classification with recursive maxima hunting
Dimensionality reduction is one of the key issues in the design of effective machine learning methods for automatic induction. In this work, we introduce recursive maxima hunting (RMH) for variable selection in classification problems with functional data. In this context, variable selection techniques are especially attractive because they reduce the dimensionality, facilitate the interpretation and can improve the accuracy of the predictive models. The method, which is a recursive extension of maxima hunting (MH), performs variable selection by identifying the maxima of a relevance function, which measures the strength of the correlation of the predictor functional variable with the class label. At each stage, the information associated with the selected variable is removed by subtracting the conditional expectation of the process. The results of an extensive empirical evaluation are used to illustrate that, in the problems investigated, RMH has comparable or higher predictive accuracy than standard dimensionality reduction techniques, such as PCA and PLS, and state-of-the-art feature selection methods for functional data, such as maxima hunting.

This paper presents an approach, called Recursive Maxima Hunting (RMH), for selecting features for functional data classification. The proposed method is shown to yield good results on simulated and real data. Although it is an adaptation of an existing method, this work remains very interesting.
This paper presents an approach, called Recursive Maxima Hunting (RMH), for selecting features for functional data classification. RMH is an adaptation of the Maxima Hunting (MH) approach, which selects features having a relevance score that is locally maximum (the distance correlation being used as relevance score). One drawback of the MH approach is that it misses the features that are not individually relevant (and are thus not local maxima) but that become relevant when combined with other relevant features. To address this problem, RMH selects the features in a recursive way, and at each stage the data are corrected by removing the information brought by the last selected feature. In practice, the method assumes that the underlying process is a Brownian motion and corrects the data by subtracting the conditional expectation of the process given the value of the last selected feature. The proposed approach was applied on 4 simulated datasets and 4 real datasets, and yields good results.  Although it is an adaptation of an existing method, this is a very interesting work. Overall, the paper is well structured and clearly written.   Some comments: - I am not sure to understand why one needs to exclude redundant points around a selected maximum (lines 15 and 20 of Algorithm 1). Isn’t the redundancy problem addressed by doing the data correction? For example, in Figure 2, one can see that the points in the close vicinity of t=5/8 have a very low distance correlation. It thus seems that one can hardly find redundant points after the correction. If it is not always the case, the authors should show an example that would help to understand why removing neighbouring points is needed. - The conditional expectation of the Brownian motion is t/t0 * X(t0) while Equation (7) says it is min(t,t0)/t0 * X(t0). If it is not a typo, the authors should discuss why they chose that correction. - It would be interesting to know the sample sizes of the real datasets.   Typos / others - Line 30: consistis —> consists - Line 72: searching from the next one —> searching for - Equation (2): two different notations are used for the distance variance. - Line 91: indepentence —> independence - Line 181: reasoble —> reasonable - Line 182: RMH algorithms —> RMH algorithm - Lines 15 and 20 of Algorithm 1: what is theta? - Lines 208-209: this sentence is not clear, since in RMH the number of selected features actually depends on some stopping criterion (i.e. the values of r and s). - Figure 4: on most x-axis labels, 50 is written instead of 1000 (for the sample size).


Efficient and Robust Spiking Neural Circuit for Navigation Inspired by Echolocating Bats
We demonstrate a spiking neural circuit for azimuth angle detection inspired by the echolocation circuits of the Horseshoe bat Rhinolophus ferrumequinum and utilize it to devise a model for navigation and target tracking, capturing several key aspects of information transmission in biology. Our network, using only a simple local-information based sensor implementing the cardioid angular gain function, operates at biological spike rate of approximately 10 Hz. The network tracks large angular targets (60
• ) within 1 sec with a 10% RMS error. We study the navigational ability of our model for foraging and target localization tasks in a forest of obstacles and show that it requires less than 200X spike-triggered decisions, while suffering less than 1% loss in performance compared to a proportional-integral-derivative controller, in the presence of 50% additive noise. Superior performance can be obtained at a higher average spike rate of 100 Hz and 1000 Hz, but even the accelerated networks require 20X and 10X lesser decisions respectively, demonstrating the superior computational efficiency of bio-inspired information processing systems.

The authors design a biologically inspired neural circuit for controlling the angle of a bat in relation to targets and obstacles detected with echolocation. The proposed neural circuit is modelled as a spiking neural network (SNN) and estimates the angle to objects. Experimental evaluation looks at spike rate in the presence of environment noise and show that a higher spike rate is not necessarily best when the environment is very noisy. The method is also compared to a PID controller on a foraging task and performs better with less computation.
I found this paper difficult to judge because I'm unsure of what the intended contribution is. From my reading of it, the authors propose a SNN model that is based on available literature of how bats track target angles. Design decisions throughout the paper are chosen to make the model more bat like. The intent is that by designing this system the authors are able to show that biologically based systems can outperform a classic control algorithm while being more computationally efficient.   In general, the technical quality of the paper is strong with several experiments, thorough analysis of them, and the method described clearly. The authors first look at a parameter of the model (spike rate) and show that a high spike rate is usually good it can be problematic when environment noise is high. They next compare their model to a PID controller for a foraging and target avoidance task. One concern I have about this experiment is that the PID controller is tuned to match characteristics of the proposed model. If the comparison is being made on % success, then the PID controller should be tuned to maximize success.   To the best of my knowledge, this is the first work that has modelled azimuth angle detection with a Spiking NN. The proposed model outperforms a PID controller for a foraging task which shows that, in this instance, biological systems can outperform a classic control algorithm.   My biggest concern with the paper is the generality of its contribution. While it is interesting that a biologically inspired network can out perform a PID controller, I'm not sure what the main contribution is aside from azimuth angle detection that mimics what is known about how bats solve this problem. If nothing more can be said about how this could relate to modelling other tasks in other animals or applying SNNs to other problems then the impact seems very low.  The paper is well written. In most places it is very clear except for some details. I'm not sure why 16 clone networks are used. The paragraph describing this (100 - 105) is vague and I can only assume it is a form of averaging although I'm not sure why this is needed. I'm also unsure about how the network parameters were chosen. There is a statement that they were chosen so the network could work with different spike rates (Line 111) but wouldn't it make more sense to find the parameters that worked best for each spike rate and use those?  Minor comments: In general the paper uses a lot of acronyms which are hard to keep in memory while reading. If the authors think this is unavoidable then it would be useful to periodically redefine them. This would enhance the readability of the paper.   [Update: I read the author's response and am satisfied with answer on how PID was tuned. After reading other reviews and author's response I raised my novelty score.]


Clustering with Same-Cluster Queries
We propose a framework for Semi-Supervised Active Clustering framework (SSAC), where the learner is allowed to interact with a domain expert, asking whether two given instances belong to the same cluster or not. We study the query and computational complexity of clustering in this framework. We consider a setting where the expert conforms to a center-based clustering with a notion of margin. We show that there is a trade off between computational complexity and query complexity; We prove that for the case of k-means clustering (i.e., when the expert conforms to a solution of k-means), having access to relatively few such queries allows efficient solutions to otherwise NP hard problems. In particular, we provide a probabilistic polynomial-time (BPP) algorithm for clustering in this setting that asks O k 2 log k + k log n) same-cluster queries and runs with time complexity O kn log n) (where k is the number of clusters and n is the number of instances). The algorithm succeeds with high probability for data satisfying margin conditions under which, without queries, we show that the problem is NP hard. We also prove a lower bound on the number of queries needed to have a computationally efficient clustering algorithm in this setting.

The paper looks at center-based clustering in Euclidean space. While it is known that the general k-means (or k-median) problem is hard in this setting, it is also known that a variety of "instance niceness" lead to poly-time clustering algorithms. The paper adds to this literature by examining the case where the instance is also "nice" *and* the algorithm has oracle access to a domain expert that is able to say whether two datapoints reside in the same cluster or not. The authors introduce an alternative notion of niceness, which they call "\gamma-margin" (Def. 1) which basically states for the every cluster center c_i, any point x in cluster i and any point y not-in cluster i we have d(y,c_i)>\gamma d(x,c_i). Then, they provide the following two results for such nice inputs: 1. (upper bound) give an efficient algorithm making O(k^2log(k) + k log(n)) oracle queries and clusters the instance perfectly. 2. (lower bound) prove that no poly-time algorithm with no access to an oracle can cluster a \sqrt{3.4}-margin instance. As a result, no poly-time algorithm can cluster a \sqrt{3.4}-margin instance with log(n) oracle calls (as one can naively traverse all possible the answers to these queries in poly-time).
This is a great paper that takes a refreshing and a novel view on an interesting line of works, and while the results are not necessarily hard technically, they are still of major significance to the field. I therefore recommend acceptance and even further - a presentation, as it can stir up an interesting discussion and lead to many follow up works. I know that I for one had many interesting follow-up questions: can such an oracle be used with other stability assumptions? what if the domain expert thinks of a clustering which is not center-based but can be well point-wise approximated by a center-based clustering? can the oracle be used to prune the tree used in [KSS04]'s O(n * exp(k)) approximation of the k-means problem?  My one major complaint is regarding the lower bound presented in the paper - which is confusing, even when one delves in the cumbersome notation. I believe that there are a few technical errors there and presentation errors that subtract from the quality of the paper and the fact that it is extremely well-written otherwise. I expect the authors' rebuttal to include answers to the following questions, as I am afraid they might lead to substantial changes to the proof of the lower bound: (1) Z_l is ill-defined (you cannot define Z_l using S_{l+1} (lines 261-262) (2) Figure 1 uses "d-\epsilon" rather than "d-\alpha" (unless I didn't understand the role of \alpha) (3) It is my understanding the L_1 is the cost of a type B cluster *per R_i* row (line 131 of appendix) and L_1-\alpha per *R_i* row with type A clustering.  And so, I believe L should include a term of L_1*l. (4) As someone who is not familiar with [Vat09], I do not see the "point" of the construction. Namely -- what should "yes" instances be mapped to? Is it to m type-A rows (indicating which sets are selected) and (l-m)-type B rows (the l-m sets not selected)? Or is it a certain selection of the B_{i,j} sets that captures which sets are taken and which aren't? (5) The construction itself is somewhat unclear. In particular, what is w? Is it \eps^{-0.5} (with \eps taken as a parameter) or some large poly(m,l) as stated in the supplementary material? I am unclear as to what is the value of k = O(n^\eps) in this construction... In other words, try to rewrite the construction s.t. it would be clear what the algorithm takes as input, and what is its output. (6) I'd appreciate it if you specifically mention the caveats in the lower bound (actually, this goes to Appendix B as well, as some constructions use Steiner points etc.). In particular, this construction places multiple points at the same location, resulting in an instance with infinite (or exponentially high if you allow miniature points perturbations), but it leaves the question of hard instances with bounded aspect ratio open...  Small comments (not to mention - nitpicking comments): * Lines 152-153 should be mentioned explicitly in the intro, or even written in an "Assumption" bulletin. * Algorithm 1 also takes as parameter \beta (or sets \beta as some sufficiently large constant, say 20.) In which case the Algorithm 1's input should be "a parameter" and not "parameters". * Can the (\gamma-1)^{-4} terms be improved to (\gamma-1)^{-2} if you use Chernoff bounds rather than Hoeffding? (i.e. multiplicative bounds on the expected value?) * Line 213 "of the oracle" => "of an oracle" * Line 217 "Section 3" (capital letter) * Line 220 "without query" => "without queries" * Suggestions: use the canonical \textsc{} to denote the X3C problem. * Section 4.1.2 - use \mathcal{X} like used at the rest of the paper. * Thm 14: as k<|X| we have that this is \log(n) effectively. Also, I am not a fan of using big-O notation in a lower bound theorem (but the theorem statement is clear).


NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization
We study a stochastic and distributed algorithm for nonconvex problems whose objective consists of a sum of N nonconvex L i /N -smooth functions, plus a nonsmooth regularizer. The proposed NonconvEx primal-dual SpliTTing (NESTT) algorithm splits the problem into N subproblems, and utilizes an augmented Lagrangian based primal-dual scheme to solve it in a distributed and stochastic manner. With a special non-uniform sampling, a version of NESTT achieves -stationary solution using O((
L i /N ) 2 / ) gradient evaluations, which can be up to O(N ) times better than the (proximal) gradient descent methods. It also achieves Q-linear convergence rate for nonconvex 1 penalized quadratic problems with polyhedral constraints. Further, we reveal a fundamental connection between primal-dual based methods and a few primal only methods such as IAG/SAG/SAGA.

The paper focuses on non-convex problems with objectives consisting of a finite sum of smooth nonconvex loss functions and a nonsmooth convex regularization. In contrast to the recent incremental algorithms, the authors propose new randomized variants of (linearized) ADMM algorithm (or called primal-dual splitting by the authors) to solve the nonconvex problem, which converge sublinearly to stationary solution. Compared to batch gradient descent, the proposed algorithm achieves better  complexity of gradient evaluations in certain cases (e.g. when Lipschitz constants are unbalanced). Some interesting connection to SAG/SAGA is revealed.
 The idea of addressing finite-sum problems through augmented Lagrangian and ADMM type algorithm is straightforward but new. Convergence results of the new randomized variation for non-convex problems are also novel, to my best knowledge. Despite the notable originality, I think a couple of places need to be further discussed or clarified.   1. Although existing work on nonconvex SVRG/SAGA algorithms [1,21] only provide analysis for smooth problems, I would presume similar results (replacing gradient by prox-gradient) can also be derived with slight modification given that the original algorithms are designed to handle nonsmooth regularization and adapted to non-uniform sampling as well. So it would be good if the author could add such comparison to nonconvex SVRG/SAGA in Table 1 when applicable.   2. It appears to me that in the NESTT-G algorithm (without reducing to the single variable form) , at each iteration, setting the remaining (N-1) x variables to z  would require O(Nd) computation and memory cost, which is not negligible. In some sense, the total computation cost could be O(N) times larger than the number of gradient evaluations. Hence, the comparison between NESTT-G and GD is not quite fair.   3. When reducing to the single variable form, the NESTT-G algorithm reduces to SAGA with special choice of stepsize. However, with uniform Lipschitz constants, the complexity of NESTT-G is worse than SAGA, Perhaps the author could explain what reason leads to the gap, and whether this can be remedied.   4. In the experiment, comparison to Gradient Descent (and non-uniform SAGA/SVRG) should be added. It would be more interesting to see results with larger size instead of just 500 data points. The notation of uniform sampling is confusing and inconsistent with the theory.   5. Minor issues: wrong relation in Thm 2.2; typo in (3.17).   == post-rebuttal answer== I have read the author's rebuttal and updated my scores accordingly.


Graphical Time Warping for Joint Alignment of Multiple Curves
Dynamic time warping (DTW) is a fundamental technique in time series analysis for comparing one curve to another using a flexible time-warping function. However, it was designed to compare a single pair of curves. In many applications, such as in metabolomics and image series analysis, alignment is simultaneously needed for multiple pairs. Because the underlying warping functions are often related, independent application of DTW to each pair is a sub-optimal solution. Yet, it is largely unknown how to efficiently conduct a joint alignment with all warping functions simultaneously considered, since any given warping function is constrained by the others and dynamic programming cannot be applied. In this paper, we show that the joint alignment problem can be transformed into a network flow problem and thus can be exactly and efficiently solved by the max flow algorithm, with a guarantee of global optimality. We name the proposed approach graphical time warping (GTW), emphasizing the graphical nature of the solution and that the dependency structure of the warping functions can be represented by a graph. Modifications of DTW, such as windowing and weighting, are readily derivable within GTW. We also discuss optimal tuning of parameters and hyperparameters in GTW. We illustrate the power of GTW using both synthetic data and a real case study of an astrocyte calcium movie.

This paper presents the problem of simultaneously aligning multiple pairs of related time series using dynamic time warping. The input to the problem consists of a set of pairs of time series, and a graph encoding a neighborhood structure on the pairs. The objective is to minimize the sum of the warping distances for each pair of time series and the sum of distances between the warping paths of neighboring pairs. The proposed solution starts from a min cut/max flow algorithm to solve standard DTW, and extends it to solve the simultaneous multiple alignment problem. An extension to learn DTW hyper-parameters using penalized likelihood-like criteria is also introduced. Results are provided for one simulated data set and one real data set.
Technical quality: The discussion of the proofs in the paper is very brief, making the technical sections quite dense. The supplemental material fills in the details and the proofs appear to be correct. One issue that could be addressed more clearly in the paper and the supplemental is the need for infinite weight reverse edges in the dual graph. This is currently not explained at all in the main paper other than stating these edges are important and later empirically demonstrating that not including them leads to violations of DTW warping constraints.   The hyper-parameter tuning section was very brief and the algorithm is not clear. As the authors are using pseudolikelihood for some of the parameters the overall approach is certainly not proper maximum likelihood estimation. The supplemental material also describes additional ad hoc modifications introduced to help with convergence of the learning approach. This section of the paper could be cleaned up substantially.  The empirical evaluation is one of the weaker aspects of the paper. The evaluation lacks depth as it relies on only one type of synthetic data set and one real data set. For the synthetic data set, a quantitative error evaluation is performed relative to independent DTW alignments in terms of comparing the estimated and true warping paths. This evaluation shows that the proposed method outperforms DTW in terms of mean error, but no standard errors or other statistics are reported to allow assessment of statistical significance of the results.   The other experiment involves and application to cell imaging data. First, this application domain needs to be described as it may not be familiar to readers. Second, the analysis of this experiment is purely qualitative. It is clear that GTW shows increased spatial coherence relative to independent DTW, but beyond this, it is not clear that anything can be concluded from the results.   Novelty/originality: The GTW problem dealt with in this paper has not been well studied in the literature. There is significant work on jointly aligning multiple curves, or alignment to a common reference, but this problem is distinct in that it concerns simultaneous alignment of multiple pairs of curves with warping functions that are allowed to be related in complex ways through the GTW graph. The use of max flow-based methods for DTW is not novel, but the authors build on it significantly to yield exact solutions to the proposed GTW problem, which is a novel contribution.  Impact: This work could have a significant impact on application areas that make use of DTW. The ability to model interactions between warping paths using arbitrary graphs is quite powerful and allows for applications to spatiotemporal domains that could only be addressed using independent DTW previously. The authors might spend some more time in the paper describing more such application domains.        Clarity and presentation: Another weakness of the paper is that the writing is not particularly clear in terms of grammar and the technical sections are quite dense. Some of the figure including 1(c) and 2(a) showing primal and dual graphs are quite hard to interpret at first.


Matrix Completion has No Spurious Local Minimum
Matrix completion is a basic machine learning problem that has wide applications, especially in collaborative filtering and recommender systems. Simple non-convex optimization algorithms are popular and effective in practice. Despite recent progress in proving various non-convex algorithms converge from a good initial point, it remains unclear why random or arbitrary initialization suffices in practice. We prove that the commonly used non-convex objective function for positive semidefinite matrix completion has no spurious local minima -all local minima must also be global. Therefore, many popular optimization algorithms such as (stochastic) gradient descent can provably solve positive semidefinite matrix completion with arbitrary initialization in polynomial time. The result can be generalized to the setting when the observed entries contain noise. We believe that our main proof strategy can be useful for understanding geometric properties of other statistical problems involving partial or noisy observations.

This paper discusses the geometrical properties of a nonconvex objective for completing a PSD matrix.  The main theorem (Thm 2.3) says all data points x s.t  grad(x) = 0 && hessian(x) >= -tau I is global minimum, this means  i)  All such x is a local minimum and globally optimal ii) All saddle points will have hessian(x) < -tau I (Later analysis shows tau <= 0.1p -- this implies the larger sampling probability, the large negative curvature the saddles have)  This coincides with the characterization of functions with only strictly ridable saddle points in previous work, hence we know some optimization algorithms e.g. cubic regularization, trust region algorithms and SGD can converge to global optima.
To prove the main theorem, the authors focus on excluding critical points with (1) large coherence with standard base (e.g Lemma 3.7) and (2) large negative curvature (e.g. Lemma 3.8) from local minima. Then they show (3) the other critical points can just be the global optimums (e.g. Lemma 3.10), this boils down to showing such points satisfies Lemma 3.3 and finally C.1, where Lemma 3.3 is reached by Lemma 3.6.  * part 2)  I think what the authors mean in Lemma 3.8 is data points satisfying 1st and 2nd order optimality conditions will have l2 norm at least ⅛, which cannot be achieved for data points whose hessian < -0.1p I. If this is true, I would suggest that the authors rephrase Lemma 3.8 for the sake of easier reading. If not, please clarify how you exclude those points.  * part 3)  I think the chain in (3) is a bit long. For better presentation, I would suggest the authors use compact this part a bit.  Besides, one needs to use Lemma C.1 to conclude that x satisfying Lemma 3.3 can only be equivalent solutions (e.g. z and -z for the rank 1 case). I think it’s better to include Lemma C.1 into main text as (a) this is an important step (b) Lemma C.1 requires an upper bound of epsilon -- a function of p in Lemma 3.3, the reader can see how p scales more clearly.   Other small details:  1) In Thm 3.2,  the authors write "tau < - 0.1p" but tau is claimed to be positive in Def 2.2. From later context (Lemma 3.4), I think the authors mean tau <= 0.1p. 2) Figure 1, the bound for l_infty norm should be 40 u / sqrt(d).  This figure is also a bit confusing. The l2 ball is actually the whole ellipsoid, not a single circle.   To summarize, I think this is a good paper, but the presentation could be improved. The proof idea could be outlined more clearly; how the technical definitions and intuitions are related to previous work could be included for readers not familiar with this area.


A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.
In the new dropout variant, we repeat the same dropout mask at each time step for both inputs, outputs, and recurrent layers (drop the same network units at each time step). This is in contrast to the existing ad hoc techniques where different dropout masks are sampled at each time step for the inputs and outputs alone (no dropout is used with the recurrent connections since the use of different masks with these connections leads to deteriorated performance). Our method and its relation to existing techniques is depicted in figure 1. When used with discrete inputs (i.e. words) we place a distribution over the word embeddings as well. Dropout in the word-based model corresponds then to randomly dropping word types in the sentence, and might be interpreted as forcing the model not to rely on single words for its task.
We next survey related literature and background material, and then formalise our approximate inference for the Variational RNN, resulting in the dropout variant proposed above. Experimental results are presented thereafter.

This paper proposes a novel approach of dropout for Recurrent Neural Networks (RNN). RNN in this paper is formulated as a Bayesian Neural Network. The authors use Variational Inference to train the RNN. They use Gaussian distributions as variational distributions for weight parameters. A novel dropout can be applied to this approach by using 0 mean Gaussian distribution as a part of variational distributions.  Previous empirical works suggested that dropout over recurrent layer was not effective. This paper suggests that bayesian approach is effective for dropout over recurrent layer. In addition, the authors apply the novel dropout to weight parameters for word embedding.  Experimental results show that this approach is effective to avoid overfitting.
Line 205: Previous method has 0 for weight decay. However, authors tune the weight decay for their proposed method. Does the proposed method outperform the previous method when the weight decay has 0? Tuning weight decay may have an enormous influence on the experimental results. I think that experimental conditions should be same except for the part of the dropout that authors propose.  Line 287-288: Why does test error decrease and then increase before decreasing again?  Figure4(a): When P_U = 0.5 and P_E = 0.0, test error can decrease after the number of epochs is 10^3. The authors should show the test error after the number of epochs is 10^3.  Figure 4(b): Why is the number of epochs less than other experiments such as Figure 4(a) and 4(c)? Figure 4(a) shows that test error drastically change after the number of epochs is 10^2. Figure 4(b) should show test error after the number of epochs is 10^2.  Figure 4(b) suggests that dropout for recurrent layer gives high test error. I think that dropout for recurrent layer is not effective even if the proposed method is applied.  In addition, Figure 4(a) and 4(b) suggest that dropout for embedding layer is more important than dropout for recurrent layer.  Other: Authors should unify the description when they refer figures in this paper. For example, “fig” is used in Line 272 but “figure”  is used in Line 285.


Data Programming: Creating Large Training Sets, Quickly
Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programmatic creation of training sets called data programming in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict. We show that by explicitly representing this training set labeling process as a generative model, we can "denoise" the generated training set, and establish theoretically that we can recover the parameters of these generative models in a handful of settings. We then show how to modify a discriminative loss function to make it noise-aware, and demonstrate our method over a range of discriminative models including logistic regression and LSTMs. Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data programming would have led to a new winning score, and also show that applying data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points over a state-of-the-art LSTM baseline (and into second place in the competition). Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.

The paper talks about a fundamental paradigm shift in building supervised ML systems. Generating ground truth (labeled dataset) to be used as training sets for different ML algorithms and frameworks through crowdsourcing or otherwise is an expensive step since it may not always be available. The data programming paradigm discussed in this paper instead focuses on how to improve existing ML systems by using unlabeled data along with some broad labeling heuristics given by domain experts (which are albeit noisy). Experimental results indicate that data programming can significantly improve supervised ML systems under certain conditions.
This is a interesting work. The author motivates the problem regarding how availability of large labeled training sets may be hindrance to several supervised ML systems and deep learning techniques. And data programming can be an interesting approach here. Also the user study indicating how researchers find it easy to write labeling heuristics instead of generating ground truth through crowdsourcing or otherwise is a good indication of the utility of this technique. The writing is clear and easy to follow, the experiments are thorough. Following are a few pointers/suggestions which I think the authors should follow up on:  1. There was an older NIPS paper by Rich Caruana regarding “Do deep nets really need to be deep” which explores the ideas of model compression. Here, a very small labeled training data is used to build a large model which then produces massive amounts of synthetically labeled dataset from massive unlabeled datasets to train compressed models. I think this should also be cited in the related work as it has been successfully deployed in many practical systems. This brings us to two significant advantages of data programming: In the former work it was mentioned that the synthetically labeled dataset needs to be around 10 times larger than the actual labeled ground truth. There are no such restrictions with respect to data programming. Second, data programming seems to improve performance of supervised ML algorithms without requiring any additional training instances which is counter-intuitive.   2. The results indicate that the coverage of the labeling functions can be as low as 7% which is quite surprising? Is their any further insight into this?  3. My one concern is that the labeling heuristics might need to have very high accuracy (as given by alpha values) which may not always be possible. It will be interesting to see what happens when labeling functions have poor to moderate accuracy even when they are independent? Nevertheless this work provides an interesting direction.