Regularized EM Algorithms: A Unified Framework and Statistical Guarantees
Latent models are a fundamental modeling tool in machine learning applications, but they present significant computational and analytical challenges. The popular EM algorithm and its variants, is a much used algorithmic tool; yet our rigorous understanding of its performance is highly incomplete. Recently, work in [1] has demonstrated that for an important class of problems, EM exhibits linear local convergence. In the high-dimensional setting, however, the M -step may not be well defined. We address precisely this setting through a unified treatment using regularization. While regularization for high-dimensional problems is by now well understood, the iterative EM algorithm requires a careful balancing of making progress towards the solution while identifying the right structure (e.g., sparsity or low-rank). In particular, regularizing the M -step using the state-of-the-art highdimensional prescriptions (e.g.,à la [19]) is not guaranteed to provide this balance. Our algorithm and analysis are linked in a way that reveals the balance between optimization and statistical errors. We specialize our general framework to sparse gaussian mixture models, high-dimensional mixed regression, and regression with missing variables, obtaining statistical guarantees for each of these examples.

In the EM algorithm for high dimensional setting, the M step is problematic, which needs to be addressed by exploiting the structure of the high-dim paramters. The paper considers to use decompposable regularizers. The main contribution is a way to set the regularizer in the iterative process of EM. It is shown that this leads to local linear convergence, under conditions about the population and empirical estimation of log likelihood using current estimation of the parameters (the basic quantity in EM). The EM algorithm is used widely in machine learning and it is thus an important topic to identify conditions under which it leads to provable guarantees. The paper gives a priciple way to regularize the EM which is particularly interesting for the high dimensional setting. It further demonstrates the application of the theory to three well known models. The result is interesting and important.--In the abstract: the authors claim that regularzing the M-step using the state of the art prescriptions ([19]) is not guaranteed to provide the desired bound, but this needs more illustration.--Condition 5: as noted by the authors, this replaces the term that appears in previous work. It seems to me that this condition is the key to make the algorithm work in the high-dim region, while the analysis follows the previous work. minor: --line 80: Y Z should be bold face --Eqn (2.2): "f" is missing --line 122: | |_R* is not yet defined here and thus the sentence is a bit confusing.


Mixed Robust/Average Submodular Partitioning: Fast Algorithms, Guarantees, and Applications
We investigate two novel mixed robust/average-case submodular data partitioning problems that we collectively call Submodular Partitioning. These problems generalize purely robust instances of the problem, namely max-min submodular fair allocation (SFA) [12] and min-max submodular load balancing (SLB) [25], and also average-case instances, that is the submodular welfare problem (SWP) [26] and submodular multiway partition (SMP) [5]. While the robust versions have been studied in the theory community [11,12,16,25,26], existing work has focused on tight approximation guarantees, and the resultant algorithms are not generally scalable to large real-world applications. This is in contrast to the average case, where most of the algorithms are scalable. In the present paper, we bridge this gap, by proposing several new algorithms (including greedy, majorization-minimization, minorization-maximization, and relaxation algorithms) that not only scale to large datasets but that also achieve theoretical approximation guarantees comparable to the state-of-the-art. We moreover provide new scalable algorithms that apply to additive combinations of the robust and average-case objectives. We show that these problems have many applications in machine learning (ML), including data partitioning and load balancing for distributed ML, data clustering, and image segmentation. We empirically demonstrate the efficacy of our algorithms on real-world problems involving data partitioning for distributed optimization (of convex and deep neural network objectives), and also purely unsupervised image segmentation.

The paper proposes new algorithms to address a set of problems falling under the umbrella term of 'submodular partitioning' -including two distinct clustering problems, namely clusteringto maximize homogeneity, or clustering so as to maximize the representation power of every cluster (e.g. so as to accelerate distributed learning). The authors consider the case where common costs are applied to each cluster (homogeneous case), or when distinct costs are applied to every cluster (hetergoeneous). Adding to this split, the authors also treat a mix of a robust and an average loss (e.g. a linear combination of the the max distortion with the average distortion when clustering).Starting from the observation that the algorithms working for the robust variants are not scalable, the authors proceed to propose (i) greedy algorithms for the robust homogeneous/heterogeneous algorithms with approximation guarantees, and (ii) a simple method for blending the solutions to the robust and the average cost optimizations.They then proceed to practical examples, demonstrating improvements over baseline algorithms for clustering. Even though the paper iswritten by experts who know how to write clearly, I found it really hard to grasp the technical part. In my understanding this is because the authors try to cover too many things (including 6 algorithms, 4 problems, and 7 theorems proved in the appendix). Even though I trust the authors know what they are doing (the appendix provides many more details, but I admit that I do not have the technical background to review them all), I think the paper would benefit from focusing on the single most interesting problem and presenting it more smoothly. The paper now reads mostly a list of pointers to other works, combinations of either existing results, or results presented in the appendix, and comes with very little explanation of the techniques being used - I was surprised that the Lovasz extension was presented.I would haveappreciated more reading (and understanding) a paper on one problem, with one algorithm, rather than a problem on 8 problems with 6 algorithms. I am also not too convinced when it comes to the evaluation of the proposed algorithms. For problems 1-3 the authors only compare to very weak baselines (random splitting). Since in all these three problems the case of lambda = 0 is considered, I would have expected that the authors either compare to the existing algorithms for robust partitioning and report either improvements in time, or in accuracy. Problem 4 is the sole case where lambda > 0 is used (which is a bit underwhelming, since treating such cases is one of the main contributions of this paper); for that problem (unsupervised segmentation) the authors again compare to weak baselines, not considering e.g. any method that uses graph-cuts. Furthermore, the authors mention in the supplemental material that their method is 'scalable' to this problem while others are not; running times are not reported, while the dependence of their method on image size is not clear. I therefore found it hard to assess how 'scalable' the method actually is. Suggestions: Apart from the broader suggestion above (to focus on a few aspects of the work), I would propose to (a) avoid coveringprior work as thoroughly, to make up space for your own contributions and (b) maybe move the 'Applications' in page 3 together with the experiments, because after the intervention of 4 pages of theory the reader has already forgotten the motivation. Minor comments: l. 259: add comma after [17] l. 304: programming -> program


Semi-Proximal Mirror-Prox for Nonsmooth Composite Minimization
We propose a new first-order optimization algorithm to solve high-dimensional non-smooth composite minimization problems. Typical examples of such problems have an objective that decomposes into a non-smooth empirical risk part and a non-smooth regularization penalty. The proposed algorithm, called SemiProximal Mirror-Prox, leverages the saddle point representation of one part of the objective while handling the other part of the objective via linear minimization over the domain. The algorithm stands in contrast with more classical proximal gradient algorithms with smoothing, which require the computation of proximal operators at each iteration and can therefore be impractical for high-dimensional problems. We establish the theoretical convergence rate of Semi-Proximal MirrorProx, which exhibits the optimal complexity bounds, i.e. O(1/ǫ 2 ), for the number of calls to linear minimization oracle. We present promising experimental results showing the interest of the approach in comparison to competing methods.

After the rebuttal:The theory developed for inexact composite mirror prox is a trivial modification of the proof of Theorem 3.1 in [10]. Authors do not hide this fact. In the end, idea-wise this paper is close to [14] and technically it does not appear to be too much involved. To me, the most interesting point of this work is that the objective function can be doubly non-smooth, in the sense f can also be non-smooth. On that front, how does this work relate tohttp://arxiv.org/abs/1502.03123A. Yurtsever, Q. Tran-Dinh, and V. Cevher "Universal Primal-Dual Proximal-Gradient Methods"which appears also to solve non-smooth constrained problems with Fenchel-type oracles?  ============ This paper proposes a new first order optimization algorithm to solve the composite problem (1) under assumptions A1-A4,S1-S3 (see the paper), which can be considered as a variant of composite mirror prox method (CMP), where the prox-operator are solved inexactly using composite conditional gradient algorithm (CCG). This paper lacks the references for some related recent works (e.g. [1] and [2]). Technically, the authors show how the inexactness in the CMP iterates accumulate in the dual gap function, they explain how we should choose the accuracy input for CCG subroutines and provide the corresponding worst case complexity analysis, which can be considered as the novel part of the work.However, from an intellectual point of view, the idea of solving prox-operator approximately reminds us [2] and it is not clear how CCG subroutine that is presented in the paper is any different from the algorithms presented in [1]. (Note that the assumption (13) is a consequence of the Holder continuity assumption in [1]).Considering the presence of [1] and [2], the significance of this work becomes questionable.  Finally, we provide below a list of minor issues and typos for the authors: 1. US English and UK English are used inconsistently. In some parts of the text 'minimization' and 'optimization' are used whereas in some other parts these are written as 'minimisation' and 'optimisation'. 2. line 91 - point is missing at line after reference [13]. 3. line 141 - 'be' missing: can 'be' much cheaper. 4. line 214 - 'of' missing: in terms 'of' this dual gap function ... 5. line 217 - typo : fro - for 6. line 247 - there is an extra parenthesis ')' at the end of line. 7. line 256 - typo: proc-mapping - prox-mapping 8. line 271 - L < inf should be L_0 < inf / or L_0 in eq (13) should be L 9. line 278 - typo: given 'on' input - given 'an' input 10. line 324 - repetition: we assume that 'we we' have 11. line 365 - Assumptions (B.1)-(B.3) should be replaced by Assumptions (S.1)-(S.3) 12. line 776 - We shall compare and 'compare' - We shall compare and 'contrast' [1] Y. Nesterov, Complexity Bounds for Primal-dual Methods Minimizing the Model of Objective Function, CORE Discussion Paper, 2015. [2] G. Lan, Gradient Sliding for Composite Optimization, arxiv:1406.0919v2.


Exploring Models and Data for Image Question Answering
This work aims to address the problem of image-based question-answering (QA) with new models and datasets. In our work, we propose to use neural networks and visual semantic embeddings, without intermediate stages such as object detection and image segmentation, to predict answers to simple questions about images. Our model performs 1.8 times better than the only published results on an existing image QA dataset. We also present a question generation algorithm that converts image descriptions, which are widely available, into QA form. We used this algorithm to produce an order-of-magnitude larger dataset, with more evenly distributed answers. A suite of baseline results on this new dataset are also presented.

Summary: This paper addresses the task of image-based Q&A on 2 axes: comparison of different models on 2 datasets and creation of a new dataset based on existing captions. Quality: The paper is addressing an important and interesting new topic which has seen recent surge of interest (Malinowski2014, Malinowski2015, Antol2015, Gao2015, etc.). The paper is technically sound, well-written, and well-organized. They achieve good results on both datasets and the baselines are useful to understand important ablations. The new dataset is also much larger than previous work, allowing training of stronger models, esp. deep NN ones.However, there are several weaknesses: their main model is not very different from existing work on image-Q&A (Malinowski2015, who also had a VIS+LSTM style model (but they were also jointly training the CNN and RNN, and also decoding with RNNs to produce longer answers) and achieves similar performance (except that adding bidirectionality and 2-way image input helps). Also, as the authors themselves discuss, the dataset in its current form, synthetically created from captions, is a good start but is quite conservative and limited, being single-word answers, and the transformation rules only designed for certain simple syntactic cases.It is exploration work and will benefit a lot from a bit more progress in terms of new models and a slightly more broad dataset (at least with answers up to 2-3 words). Regarding new models, e.g., attention-based models are very relevant and intuitive here (and the paper would be much more complete with this), since these models should learn to focus on the right area of the image to answer the given question and it would be very interesting to analyze the results of whether this focusing happens correctly. Before attention models, since 2-way image input helped (actually, it would be good to ablate 2-way versus bidirectionality in the 2-VIS+BLSTM model), it would be good to also show the model version that feeds the image vector at every time step of the question.Also, it would be useful to have a nearest neighbor baseline as in Devlin et al., 2015, given their discussion of COCO's properties. Here too, one could imagine copying answers of training questions, for cases where the captions are very similar.Regarding a broader-scope dataset, the issue with the current approach is that it is too similar to the captioning approach or task, which has the drawback that a major motivation to move to image-Q&A is to move away from single, vague (non-specific), generic, one-event-focused captions to a more complex and detailed understanding of and reasoning over the image; which doesn't happen with this paper's current dataset creation approach, and so this will also not encourage thinking of very different models to handle image-Q&A, since the best captioning models will continue to work well here. Also, having 2-3 word answers will capture more realistic and more diverse scenarios; and though it is true that evaluation is harder, one can start with existing metrics like BLEU, METEOR, CIDEr, and human eval. And since these will not be full sentences but just 2-3 word phrases, such existing metrics will be much more robust and stable already.Some other comments:-- Sec 4.2.1: The authors say splitting happens on conjunctions to give 'two independent sentences', but what if these creates a sentence and a clause, instead of two sentences? -- Sec 4.2.1: Indefinite determiners "a(n)" to definite determiners "the": what if the caption had something plural without an 'a', e.g., "two boys are playing in the park" and the question still needs a 'the', i.e., "where are the two boys playing?" -- Sec 5.5: IMG+BOW works well and the authors discuss the reasons for this to be the noisy dim-reduction of 4096-dim image vectors, and that the sequential RNNs are not needed for Q&A. However, this needs to ablated better to know which of these 2 reasons is more true, e.g., by having a IMG+BOW baseline on 300-500-dim image vectors instead of 4096-dim, and/or by using 4096-dim word embeddings (which should most likely be similarly accurate to 500-dim ones) in RNNs with the 4096-dim image vector.References: Exploring Nearest Neighbor Approaches for Image Captioning Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, C. Lawrence Zitnick Clarity: The paper is well-written and well-organized.Originality: The task of image-Q&A is very recent with only a couple of prior and concurrent work, and the dataset creation procedure, despite its limitations (discussed above) is novel. The models are mostly not novel, being very similar to Malinowski2015, but the authors add bidirectionality and 2-way image input (but then Malinowski2015 was jointly training the CNN and RNN, and also decoding with RNNs to produce longer answers).Significance: As discussed above, the paper show useful results and ablations on the important, recent task of image-Q&A, based on 2 datasets -- an existing small dataset and a new large dataset; however, the second, new dataset is synthetically created by rule-transforming captions and only to single-word answers, thus keeping the impact of the dataset limited, because it keeps the task too similar to the generic captioning task and because there is no generation of answers or prediction of multi-word answers.


Kullback-Leibler Proximal Variational Inference
We propose a new variational inference method based on a proximal framework that uses the Kullback-Leibler (KL) divergence as the proximal term. We make two contributions towards exploiting the geometry and structure of the variational bound. First, we propose a KL proximal-point algorithm and show its equivalence to variational inference with natural gradients (e.g., stochastic variational inference). Second, we use the proximal framework to derive efficient variational algorithms for non-conjugate models. We propose a splitting procedure to separate non-conjugate terms from conjugate ones. We linearize the non-conjugate terms to obtain subproblems that admit a closed-form solution. Overall, our approach converts inference in a non-conjugate model to subproblems that involve inference in well-known conjugate models. We show that our method is applicable to a wide variety of models and can result in computationally efficient algorithms. Applications to real-world datasets show comparable performances to existing methods.

The paper presents a general method for non-conjugate variational inference based on proximal method and linearisation of the non-conjugate model. This is shown to reduce to natural gradient optimisation for conjugate exponential models. The method is shown to lead to slightly better predictive accuracy than standard approximate inference methods in a few selected problems and data sets.      Quality      The method relies on linearisation to handle non-conjugate models. The seems potentially problematic, as previous works have found linearisation to be unreliable in variational inference with non-conjugate models (see e.g. Honkela and Valpola, NIPS 2004). The method is evaluated empirically on a few data sets, and mostly found to perform well. The paper focuses a lot on the positive aspects of the method, and its weaknesses and limitations are not mentioned at all.      Clarity      The paper is mostly clearly written and well-organised, but some key details are missing (specifically: what exactly is f_n).      Originality      The proposed approach is novel, although it is based on combination of existing techniques from various fields. Use of references is a bit shaky: the good fundamental reference from proximal algorithms seems to be missing and furthermore previous approaches using linearisation with non-conjugate variational inference are not discussed.      Significance      As the reliability of the proposed method is questionable, it is difficult to judge the paper's importance until this is resolved.      Other comments      1. The last sentence of the abstract contains unsubstantiated advertising: you method is not the best in every way. This must be clarified or deleted.      2. In Eqs. (2)-(3), what is eta?      3. In Eq. (3), why is it arg min and not arg max as in Eq. (2)?      4. As the proposed method seems to be a purely batch algorithm, it is unclear why it cites and links to SVI methods so much. Classic batch VB would seem much more relevant here.      5. You should clarify what f_n is precisely in the examples you study. Also, the accuracy of the linearisation procedure needs to be checked to make sure you do not suffer from similar problems as illustrated in Fig. 1 of Honkela and Valpola (NIPS 2004) for the Taylor series linearisation.      Additional references      Previous use of linearisation in variational inference:     Lappalainen, Harri, and Antti Honkela. "Bayesian non-linear independent component analysis by multi-layer perceptrons." In Advances in independent component analysis, pp. 93-121. Springer London, 2000.      Analysis of problems with linearisation:     Honkela, Antti, and Harri Valpola. "Unsupervised variational Bayesian learning of nonlinear models." In Advances in neural information processing systems, pp. 593-600. 2004.


Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
This paper presents a new semi-supervised framework with convolutional neural networks (CNNs) for text categorization. Unlike the previous approaches that rely on word embeddings, our method learns embeddings of small text regions from unlabeled data for integration into a supervised CNN. The proposed scheme for embedding learning is based on the idea of two-view semi-supervised learning, which is intended to be useful for the task of interest even though the training is done on unlabeled data. Our models achieve better results than previous approaches on sentiment classification and topic classification tasks.
like phrases, or regions of size 20 like sentences), eliminating the extra layer for word vector conversion. This direct learning of region embedding was noted to have the merit of higher accuracy with a simpler system (no need to tune hyper-parameters for word vectors) than supervised word vector-based CNN in which word vectors are randomly initialized and trained as part of CNN training. Moreover, the performance of [11]'s best CNN rivaled or exceeded the previous best results on the benchmark datasets.
Motivated by this finding, we seek effective use of unlabeled data for text categorization through direct learning of embeddings of text regions. Our new semi-supervised framework learns a region embedding from unlabeled data and uses it to produce additional input (additional to one-hot vectors) to supervised CNN, where a region embedding is trained with labeled data. Specifically, from unlabeled data, we learn tv-embeddings ('tv' stands for 'two-view'; defined later) of a text region through the task of predicting its surrounding context. According to our theoretical finding, a tv-embedding has desirable properties under ideal conditions on the relations between two views and the labels. While in reality the ideal conditions may not be perfectly met, we consider them as guidance in designing the tasks for tv-embedding learning.
We consider several types of tv-embedding learning task trained on unlabeled data; e.g., one task is to predict the presence of the concepts relevant to the intended task (e.g., 'desire to recommend the product') in the context, and we indirectly use labeled data to set up this task. Thus, we seek to learn tv-embeddings useful specifically for the task of interest. This is in contrast to the previous word vector/embedding learning methods, which typically produce a word embedding for general purposes so that all aspects (e.g., either syntactic or semantic) of words are captured. In a sense, the goal of our region embedding learning is to map text regions to high-level concepts relevant to the task. This cannot be done by word embedding learning since individual words in isolation are too primitive to correspond to high-level concepts. For example, "easy to use" conveys positive sentiment, but "use" in isolation does not. We show that our models with tv-embeddings outperform the previous best results on sentiment classification and topic classification. Moreover, a more direct comparison confirms that our region tv-embeddings provide more compact and effective representations of regions for the task of interest than what can be obtained by manipulation of a word embedding.

The paper presents an approach to learn features for region representation from unlabeled data, for use in text classification. The paper shows that using unlabeled data to estimate predictive features for text regions in a two-view learning framework brings substantial improvements over purely supervised learning of representations, and that learning region embeddings is substantially more effective than using word-embeddings (such as ones produced via word2vec).The paper explores different learning targets from unlabeled data as well as different representations of regions (bag-of-words vs concatenation). It is shown that the resulting representations are complementary and the combination is very effective.The main lesson from the paper: that learning region embeddings on unlabeled data is substantially more effective than the commonly employed method of using word-embeddings on unlabeled data and relying solely on them to represent regions, has substantial implications for NLP research.The paper is written very well and is easy to follow. The approach is original. The experiments are very thorough and bring insight into the performance of the different options.Improvements over the previous state-of-the-art obtained by use of deeper CNN models orsemi-supervised n-gram models are substantial and convincing.


Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion. At each level of the pyramid, a separate generative convnet model is trained using the Generative Adversarial Nets (GAN) approach [11]. Samples drawn from our model are of significantly higher quality than alternate approaches. In a quantitative assessment by human evaluators, our CIFAR10 samples were mistaken for real images around 40% of the time, compared to 10% for samples drawn from a GAN baseline model. We also show samples from models trained on the higher resolution images of the LSUN scene dataset.

As in the summary -- the paper implements a sensible idea, it is clearly written, and the generated images are beautiful. I have a concern that the quantitative results are misleading and/or wrong.I have a suspicion that the model may be performing similarly to a computer graphics engine ... where it generates very good naturalistic images, but where most real images would be assigned extremely low probability. This could make for a fine paper, but the results would need to be presented in a way that makes this clear.Detailed comments as follow:45 - 'indicating a better density model...' don't think this part follows57-61 - Should note these aren't actually generative models as typically defined. Though p(visible|hidden) is straightforward, p(hidden) is complex and difficult to sample from.72 - 'much more ambitious', and/or addresses a different goal93 - 'their respective' -> 'both'148 - prob. move footnote to main text, following a comma at the end of the sentence186 - 'can directly' -> 'can be directly'191-193 - Is this true? I would imagine fitting a model to each scale would just provide more parameters with which to overfit.220 - I wonder if it will be difficult for these variables to represent something interpretable about the image, since there is only one noise variable per pixel, and the noise variables are interpreted convolutionally which will largely constrain them all to play a very similar role.226 - On a validation set?231 - Good idea + interesting to visualize variation captured.239 - 'nature image' -> 'natural image'Table 1: - Give units. Nats/image?- If the units are nats/image, then I believe this performance would be state of the art if you compare against lossless image compression of 24-bit full size images. This would be impressive! But it also makes me extremely skeptical, because my understanding is that Parzen window techniques badly undershoot the true log likelihood, and 50,000 samples is a very small number of samples to fill the space of 32x32 images -- or even to fill a state space of dimensionality (32x32*3 - 16x16*3), after the conditioning. (e.g., assuming pixel values are integers, you would need order exp(1799 * (32*32*3 - 16*16*3) / (32*32*3)) samples to accurately capture the distribution using your Parzen window technique ... which is so much larger than 50000 that it leads to a floating point overflow when I try to compute it)- Did you add uniform noise in [-0.5, 0.5] to each pixel value in your training and test sets? If not, it may be that scale 0 is learning to capture something about the integer quantization of pixel intensities. This is cheating, in that you can get infinite likelihood without learning anything about natural images by modeling the data as delta functions at integer pixel intensities. This could matter even for the multiscale Parzen approach, since it may be possible to exactly predict pixel values at scale 0 conditioned on scale 1.- I'm especially suspicious that Parzen is being performed incorrectly, because you are also using non-comparable Parzen variants to compare GAN and LAPGAN. See Appendix A comment below.- (A really neat followup project, if the numbers do turn out to be real, would be to use something similar to this method for image or video compression. I think the multi-scale structure would make for a straightforward encoding and automatically get you smooth performance degradation with limited bandwidth.)- There are other models you could compare against here. [24] provides a bound on log likelihood for CIFAR-10. (and I believe also a deep learning workshop paper at this year's ICML -- will try to go back and look this up)(Update: I realized from reviewer conversation that these log likelihoods are for 28x28 pixel images, not 32x32. That does explain the difficulty in direct comparison, and makes the log likelihood about 25% more believable. It would be good to note the image size in Table 1.)Figure 5. - This is both interesting and difficult to know how to interpret. It seems as if image generation is quite stereotyped given initial 4x4 seeds. That would be consistent with the noise variables entering into later layers in a possibly difficult to utilize way (line 220). One interpretation might be that the top layer GAN creates most of the diversity in the generated samples, and that the conditional layers fill in consistent but inflexible details.Section 5.3: - It's very difficult to know how subjects will choose to label something as "real" or "not real". For instance, if an image appears out of focus, or pixelated, or like an animal that you can't quite discern, do they still label it real? If you want to do a fair comparison against natural images, it would be much, much better to force subjects to make a choice between generated images and natural images. (this would be a "two alternative forced choice" experiment, and is extremely common)- Who were the volunteers? What instructions did they receive? Did they know when they were looking at GAN vs. LAPGAN images, or what type of responses the experimenters hoped for in each session? Was the experiment single blind? Double blind? Did the subjects have previous experience with all three datasets? Strangers on mechanical turk, and a double blind experiment, would be better.Some suggestions for two possible improved psychophysics experiments: 1) Do generated images look real: Show two images next to each other, and ask the user to select which one is 'real'. As they approach chance performance, this will be a strong indication that the images produced by the model appear realistic. 2) Are generated images diverse: Generating realistic images doesn't mean that you've captured the distribution over real images. For instance, maybe all the images you generate look very similar. One experiment that could significantly address this is to show real and generated images next to each other. Then show a third image, either real or generated, below, and ask which of the first two the third image is most similar to. The degree to which generated images are chosen to be similar to each other at above chance level is a measure of their diversity relative to natural images.- If you are interested in the effect of exposure time, you should display a mask image (e.g. checkerboard or white noise) after the stimulus, rather than a gray screen. Otherwise your subject will have additional time to view the afterimage.- You should provide a fixation target (e.g. a small cross) before each trial, and instruct the subject to fixate on it.- This experiment seems obviously harmless, but you should be *very careful* about doing human experimentation without approval from an institutional review board. Running human experiments without review by an ethics board can get you in a lot of trouble, especially in a university setting. Many institutions have short online forms and a very easy expedited process for obviously harmless experiments like this one. Getting approval should be pretty lightweight, but is strongly recommended.403 - 'previous scales' -> 'more than one previous scale' (assuming I've understood what's going on correctly)Appendix A: This is a neat idea! Nice.Parzen window estimates tend to be biased low because they require an exponential number of samples to fully explore a high dimensional space. Your method instead requires samples to explore a sequence of lower dimensional spaces. This will make it much easier for the samples to explore the state space, and can be expected to greatly increase the Parzen log likelihood, which is great.You measure the GAN log likelihood using a Parzen estimate in the full space, and the LAPGAN log likelihood using your modified Parzen estimate. This isn't a fair comparison, since it is impossible to tell how much of the improvement is due to the improved Parzen estimation method, and how much is due to better performance of the model.I strongly encourage you to include a third row in Table 1, where you give the LAPGAN log likelihood computed with the same Parzen estimator as the GAN row. Then it will be possible to tease apart model improvement vs. measurement technique improvement.417 - notation, should probably make this q_0(h | l)--After reading the comments from Assigned_Reviewer_2, I share their concern that the number of latent variables is smaller than the number of free data dimensions in the conditional GAN layers (by a factor of 3 I believe). I would also like to request clarification on this.


Matrix Manifold Optimization for Gaussian Mixtures
We take a new look at parameter estimation for Gaussian Mixture Model (GMMs). Specifically, we advance Riemannian manifold optimization (on the manifold of positive definite matrices) as a potential replacement for Expectation Maximization (EM), which has been the de facto standard for decades. An out-of-the-box invocation of Riemannian optimization, however, fails spectacularly: it obtains the same solution as EM, but vastly slower. Building on intuition from geometric convexity, we propose a simple reformulation that has remarkable consequences: it makes Riemannian optimization not only match EM (a nontrivial result on its own, given the poor record nonlinear programming has had against EM), but also outperform it in many settings. To bring our ideas to fruition, we develop a welltuned Riemannian LBFGS method that proves superior to known competing methods (e.g., Riemannian conjugate gradient). We hope that our results encourage a wider consideration of manifold optimization in machine learning and statistics.

This paper revisits fitting mixtures of Gaussians with gradient methods. A combination of manifold optimization, with a particular linesearch procedure, and a "reparameterized" objective, empirically beats batch EM in the experiments reported.While I'm not an expert in Riemannian manifolds, the work appears plausible. The work is a reported at a level that's possible to follow, if taking results from cited references on trust.Plain mixtures of Gaussians may not seem particularly exciting in this world of Bayesian nonparametrics and deep neural networks. However, GMMs are quietly plodding along as important work-horses in applications: they're useful and surprisingly powerful. People often then want to extend GMMs in various ways, e.g., by adding constraints or tying covariances together. But then EM usually doesn't apply. Gradient-based procedures would generalize fine, so ones that work well on the base model could have significant impact on statistical applications.I would previously have used an unconstrained parameterization based on the Cholesky. While the paper gives reasons to disfavour this approach, it's a shame it's not included in the empirical comparison. Are spurious stationary points an issue in practice? Is the manifold optimizer actually better? Is the main gain here from the "representation" in (2.3); would that also help the Cholesky approach? The paper would be stronger if it answered these questions.  Minor comments:Line 081: "open a new" -> "open new"Line 138: rely *on* geodesicsThe timings in Table 4 would be probably be better as a graph. A graph would give an idea of the comparison more quickly. Precise times are irrelevant, being implementation and machine dependent.I suggest replacing lines 147-149 "At any point... specifically [4]" with: "Geodesics on the manifold from $\Sigma_1$ to $\Sigma_2$ are given by [4]:". Riemannian metrics are not explained in the paper, and don't need to be understood or mentioned to take on trust the geodesic result. (The lack of context, and the nearby definition of `$d$', encourges a misreading of $d$ as a variable, making the trace expression evaluate to `$d^3$' (!).)I suggest adding the zero mean ("0,") explicitly to the normal density in (2.3) and the line below it. On a quick reading, augmenting the data with a constant in the line above (2.3) seems like a terrible idea, a Gaussian could fit these new data with infinite density. The zero-mean constraint is what stops that from happening. Also, the compact Gaussian notation is only defined explicitly including a mean term above (2.1).I wouldn't call the transformed problem (2.3) a "reparameterization". A different model is being fitted, one that would generate data from a different distribution than the original.I suggest numbering all equations so other people can refer to them, not just in reviews, but in reading groups and citations.


Fast Second-Order Stochastic Backpropagation for Variational Inference
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametrization trick with lower complexity. As an illustrative example, we apply this approach to the problems of Bayesian logistic regression and variational auto-encoder (VAE). Additionally, we compute bounds on the estimator variance of intractable expectations for the family of Lipschitz continuous function. Our method is practical, scalable and model free. We demonstrate our method on several real-world datasets and provide comparisons with other stochastic gradient methods to show substantial enhancement in convergence rates.

This paper presents a scalable second-order stochastic variational inference method for models with normally distributed latent variables. In order to efficiently compute the gradient and Hessian, a representrization trick for general location-scale family is adopted with computation scales quadratically w.r.t the number of latent variables for both gradient and Hessian (in practice with diagonal Gaussian). Furthermore, Hessian-free optimization is used to account for the high dimensionality of the underlying embedded parameters. R-operator technique is used and it enables exact Hessian-vector product computation in Hessian-free optimization. L-BFGS can also be used in place of Hessian-free within the proposed framework. On the theoretical side, the paper provides a variance bound which explains why in some earlier work even only one sample from the variational distribution is sufficient to construct a noisy gradient with low variance. Experimental results are reported on both Bayesian logistic regression and variational autoencoder. The proposed Hessian-free or L-BFGS based methods understandably converge faster and to better local optima. **Quality**Overall the paper is of good quality with both practical and theoretical contributions. Deriving bounds is not my expertise, so I couldn't follow all of the proof for Theorem 2 in the supplementary material. But as far as I can understand, the derivation and proof seem sound. The evaluation clearly demonstrates the advantage of the proposed methods. **Clarity**This paper seems to be written in a hurry with some parts not clear enough. Some of the descriptions are vague (mostly in the Section 3 and 4) and probably difficult to understand without enough context - I have to interpolate the authors' intention sometimes when reading the paper. I would suggest the authors carefully proof-reading the paper and filling in more context if possible. Detailed comments:1. Line 113: Presumably g and H denote the gradient and Hessian of f(z) respectively, but it was never formally defined in the main paper (it is defined in supplementary material).2. Line 128: leading to an algorithmic -> (based on supplementary material) leading to an algorithmic complexity3. Line 199: none-the-less -> nonetheless4. Supplementary material Line 160: lawyer-wise -> layer-wise5. Supplementary material Line 250: Should be Eq. (10) not Eq. (11).6. For the second part of the experiment, it is not clear to me why there is running time/objective reported for both train and test. From what I understand, the variational auto-encoder is trained on the training data to learn the parameters for both the encoder and decoder networks with the proposed and competing methods. For test, you feed the test images as the input to the encoder network and reconstruct them from the output of the decoder network (as I understand this is how the images in Figure 2(b) and 3(b) are generated, the authors didn't mention anything about how they are generated). Does the fact that there is also a running time for test mean that the test data is also used to train the model? I hope not, otherwise the reconstruction results would not be reliable. We just need more context to understand the evaluation. If the space is a concern, I would suggest removing some of the introduction to BFGS and some background stuff in Section 3.**Originality**The proposed method is built on top of the work on stochastic gradient variational Bayesian (SGVB), which mostly only uses first-order method with adaptive learning rate tuning mechanism (e.g. AdaGrad/RMSProp). Incorporating second-order information isn't something groundbreaking but the paper presents a way to efficiently compute the gradient and Hessian, which is still fairly valuable. Furthermore, Hessian-free and L-BFGS can be applied and benefit from the efficient gradient computation. The theoretical analysis on the variance bound is very exciting work because it provides the theoretical justification for something that has been observed empirically. **Significance**The evaluation is reasonably thorough. Given that there isn't new algorithm/modeling innovation in this paper, reporting optimization objective v.s. time is the most convincing way to demonstrate a better optimization strategy. I do have one comment on the Bayesian logistic regression experiment and would like the authors to investigate and maybe address it in the supplementary material (given that the space seems already a problem for the main paper): Looking at the coefficients in Figure 3 of the supplementary material, the magnitude of DSVI is orders of magnitude larger than the proposed methods for the first two datasets. Is there a particular reason behind this? On the much bigger A9a dataset, all three methods learn coefficients which are in the same scale.


Copeland Dueling Bandits
A version of the dueling bandit problem is addressed in which a Condorcet winner may not exist. Two algorithms are proposed that instead seek to minimize regret with respect to the Copeland winner, which, unlike the Condorcet winner, is guaranteed to exist. The first, Copeland Confidence Bound (CCB), is designed for small numbers of arms, while the second, Scalable Copeland Bandits (SCB), works better for large-scale problems. We provide theoretical results bounding the regret accumulated by CCB and SCB, both substantially improving existing results. Such existing results either offer bounds of the form O(K log T ) but require restrictive assumptions, or offer bounds of the form O(K 2 log T ) without requiring such assumptions. Our results offer the best of both worlds: O(K log T ) bounds without restrictive assumptions.

This paper tackles the dueling bandits problem, focusing on the scenarios where a Condorcet winner does not exist. In particular it focuses on finding the Copeland winner (i.e., the arm that beats maximal number of other arms) and proposes two algorithms for this task. The paper provides theoretical bounds on these algorithms which indicate better asymptotic regret performance than previous approaches for identifying the Copeland winner.Overall the paper (with the supplementary material) was quite solid in terms of material. However I did have some significant concerns.I wanted to start my critique with the informativeness and "quality" of the actual 9 page material. I recognize that this is a matter of principle to which the authors may object, but in my honest opinion I would like to reward the authors and NIPS submissions that took the extra effort of making a paper self-contained and informative within the 9 page limit, although that may have meant leaving out certain interesting results/analysis/experiments. At the end of the day what gets published is the 9 pager, which is what I believe should be the determining factor of a paper's quality. After all the NIPS guidelines clearly state that I reserve the right to do so:"Note that the reviewers and the program committee reserve the right to judge the paper solely on the basis of the 9 pages of the paper; looking at any extra material is up to the discretion of the reviewers and is not required."With this in mind, I actually find the overall quality and clarity of the 9 pager itself fairly lacking. Most of the interesting analysis and all the other empirical evidence are hidden away in the 20+ page appendix. In fact the overall analysis of the algorithms itself is fairly hard to understand as the 9-pager itself presents it in a rather hard-to-understand manner. Most of the paper constantly makes forward references to the supplementary material to complete the story, though its' purpose is to "supplement" the main content not serve as a continuation of it.Some of the things which I would have liked to see in the actual 9-pager itself would include:a) Significance: Why is this work significant? Lately the work on dueling bandits is becoming increasingly incremental with different approaches improving marginally over the previous on different metrics. So why is the Copeland winner what we should really be interested in as compared to the Borda winner or the von Neumann winner or some yet-to-be-published winner? Because at the end of the day what a practitioner cares about is the method that works best in practice not the specific metric that was used to get to that method. Towards this, I don't really find this paper providing any compelling evidence that the Copeland winner is the right one to aim for.b) (Quality+Significance) Related to the above, but why would Condorcet methods not work well in cases such as this (even though the Condorcet winner may not exist). The authors mention that these methods may fail for these problems, but the 9 pager itself does not contain any evidence (theoretical or empirical) indicating the same.c) (Quality+Clarity) Providing empirical evidence in the main material (even a small study whose setup can be detailed in the appendix) could convince a practitioner to switch such a method. A set of involved hard-to-gauge theoretical results alone do not constitute a compelling enough set of evidence in my opinion.Again while I understand that the authors have addressed some of these criticisms in the supplementary material, I am basing this review on the 9 page material to be fair to all the other papers I review which have relied on me reading just the 9 pages and not a 32 pager.


Fast and Guaranteed Tensor Decomposition via Sketching
Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are unique to tensors. We develop novel methods for randomized computation of tensor contractions via FFTs, without explicitly forming the tensors. Such tensor contractions are encountered in decomposition methods such as tensor power iterations and alternating least squares. We also design novel colliding hashes for symmetric tensors to further save time in computing the sketches. We then combine these sketching ideas with existing whitening and tensor power iterative techniques to obtain the fastest algorithm on both sparse and dense tensors. The quality of approximation under our method does not depend on properties such as sparsity, uniformity of elements, etc. We apply the method for topic modeling and obtain competitive results.

# Summary The paper applies existing tensor sketching ideas to orthogonal tensor decomposition, leading to an algorithm with running time O(n^2 log(n)) (vs. O(n^3)), when the eigenvalue spectrum decays superlinearly. The authors propose a new hashing method for symmetric tensors, and validate their computational speedups on topic models.# Comments * The idea of storing a low-dimensional sketched representation is exciting, however the results presented in Theorem 2 only hold when the spectrum of the tensor (the eigenvalues) decay super-linearly, as in the synthetic experiments (e.g. $\lambda_i = 1/i$). In the case where the spectrum is flat, i.e. $\lambda_i$ are approximately the same, the size of the sketch required is $O(n^3)$, according to Theorem 2. Is this an artifact of the analysis or a limitation of the approach? * In the experiments, and in table 3, the sketch building time is ignored -- what exactly comprises of sketch building time? It seems like this is a crucial component to the method. * How does the method scale with the magnitude of error ($\sigma$)? * Is the sketching of a rank-1 tensor (3.2.1) a contribution of the current paper? It appears to be presented in [22; Algorithm 1]. * How does the symmetric tensor sketch proposed in 3.4 differ from the idea of using the complex ring to count higher moments, e.g. Ganguly, Sumit. "Estimating frequency moments of data streams using random linear combinations." Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques. Springer Berlin Heidelberg, 2004. 369-380.# Minor commments * The notation in the paper is hard to follow and very densely packed. In some places, the notation is not clearly defined, e.g. 3.1., [n] was not defined, and $p$ is not described. * In Algorithm 2, the footnote 4 is easily confused with an exponent. Also, it is not clear whether line 7 has a typo: "$[v^{(m)}]_i \gets \ksi_1(i) ..$" <- is it always $\ksi_1$? * What is meant by time constraints when describing the synthetic experiments (margin line #392).


Scalable Semi-Supervised Aggregation of Classifiers
We present and empirically evaluate an efficient algorithm that learns to aggregate the predictions of an ensemble of binary classifiers. The algorithm uses the structure of the ensemble predictions on unlabeled data to yield significant performance improvements. It does this without making assumptions on the structure or origin of the ensemble, without parameters, and as scalably as linear learning. We empirically demonstrate these performance gains with random forests.

This paper focusses on developing an efficient transductive ensemble learning algorithm that follows the theoretical work of [1]. The problem, that is characterized as a zero-sum game between a predictor and nature, is first reviewed. The authors then introduce a generalization to the scenario where the voters of the ensemble are "specialists" : voters that can abstain on a subset of the examples. An instanciation of the method to specialists learned using a Random Forest (named HedgeClipper) is devised from the theoretical framework, and empirically evaluated on real world datasets.This paper is very well written and its original contributions are made very clear by the authors. The introduction and preliminaries are very complete, allowing the reader to understand the work of [1]. While tackling two out of four open problems stated in [1], the presented learning algorithm seems to be efficient, versatile, and fills in the gap between the theoretical framework of [1] and the practice.The generalization to so-called "specialists" is interesting. The resulting learning algorithm based on Random Forests seems to be one of many ways to benefit from the introduced ideas, and open the way the new areas of investigation. To conclude, this is a relevant generalization and instanciation of the novel theoretical work of [1]. I am unsure about the novelty aspects, but the work seem original and significant.  Typos and other minor comments : - Reference [5] appears before Reference [4], which should be correct by simply re-compiling. - Line 32: therefor -> therefore - Figure 1 is a bit blurry (on paper), and the {+1, -1} cells are not distinguishable when printed on paper. - Figure 2 is also a bit blurry. - Footnote numbers in the text have an extra space (before the number). - The source code is conveniently based on Scikit-Learn, and should be made publicly available after publication. - Line 514 of Supplementary Material has an extra space before the reference to Section 3.2.  [1] Akshay Balsubramani and Yoav Freund. Optimally combining classifiers using unlabeled data. In Conference on Learning Theory, 2015. To appear. Update after rebuttal: I kept the same quality score.


Rate-Agnostic (Causal) Structure Learning
Causal structure learning from time series data is a major scientific challenge. Extant algorithms assume that measurements occur sufficiently quickly; more precisely, they assume approximately equal system and measurement timescales. In many domains, however, measurements occur at a significantly slower rate than the underlying system changes, but the size of the timescale mismatch is often unknown. This paper develops three causal structure learning algorithms, each of which discovers all dynamic causal graphs that explain the observed measurement data, perhaps given undersampling. That is, these algorithms all learn causal structure in a "rate-agnostic" manner: they do not assume any particular relation between the measurement and system timescales. We apply these algorithms to data from simulations to gain insight into the challenge of undersampling.

This paper proposes a framework and algorithms for learning causal structures from "undersampled" time series data, i.e. where the sampling occurs at a lower rate than the system, when it is unknown what the ratio of undersampling is; the only previous approach to deal with the undersampling problem required that the ratio of undersampling be small and known.The authors attempt to learn an equivalence class of structures for which an undersampling ratio exists that is consistent with the observed time series by learning a structure for the time series and then iterating over possible structures for the system while pruning the solution tree. The authors prove that their first algorithm, RASL_{re} is correct and complete and then introduce more efficient refinements, RASL_{ie} and RASL_{il}. The authors devote a significant portion of the paper to simulation studies comparing these three algorithms in different contexts where the structure for observed time series is known and then the authors provide one synthetic data example when this structure is unknown. The authors are dealing with a very difficult problem that is often shoved under the rug in related approaches. The framework for how they've approached this problem is very interesting and may turn out to be very fruitful, but the paper appears to be very preliminary at this point and is missing some things. The results section focuses a lot of attention to comparing the 3 different versions of RASL when the correct structure for the undersampled time series is known, which is essentially an implementation issue, at the expense of testing the full procedure in a realistic context when this graph is unknown. Only one synthetic data example is given with a 5 node structure and the discussion is limited. I'm left not really knowing what the significance of the edge omission and commission rates are, whether they converge, or how scalable this algorithm. The lack of discussion is sort of a problem throughout the results section and exacerbated by the fact that there is no conclusion. There are a lot of interesting observations given, but not much in terms of holistic conclusions about the performance of the algorithm. Aside from the last two sentences in 4.2, the authors don't really do much to "sell" their procedure. The authors also neglect the issue of learning the structure of the undersampled system. Several methods are mentioned, but the authors do not discuss the assumptions or implications of using one of these methods over another or whether they are correct or complete (which seems important since the authors prove their method which uses the results from these methods is correct and complete). Results are only given SVAR because the authors say it gave accurate and stable solutions in pilot testing, but they don't discuss further as to why they picked SVAR and whether they tested other methods. Including results for these other methods is another way that the results section could be improved. The authors also mention a number of possible application areas at the beginning of the paper, but do not apply their method to any real data. Some attempt to assess the performance on real data (even if the absolute ground truth is unknown), would also improve the results section.


Equilibrated adaptive learning rates for non-convex optimization Harm de Vries
Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the socalled equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

This paper proposes a new adaptive learning rate scheme for optimizing nonlinear objective functions that arise during the training of deep neural networks. The main argument is based on recent results that indicate that the difficulty of the optimization stems from the presence of saddle points rather than local minima in the optimization path.The saddle points slow down training since the objective function tends to be flat in many directions and ill-conditioned in the neighbourhood of the saddle points. The authors propose a new method for reducing the ill-conditioning (the problem of pathological curvature) by "preconditioning" the objective function through a linear change of variables, which reduces to left-multiplying the gradient descent update step with a learned preconditioning matrix D. They focus specifically on the case where D is diagonal, and they show how a diagonal D reduces to methods for learning parameter-specific learning rates, such as the well-known Jacobi preconditioner or RMSProp.  This is a nice framework within which to consider different schemes for adaptive learning rates. As a side note, it was unclear to me whether this link has been established before, or whether this was a contribution of this paper?Their main *theoretical* contribution is to show that the Jacobi preconditioner has undesirable behaviour in the presence of both positive and negative curvature, and to derive the optimal preconditioner which they call the "equilibration preconditioner" D^E as |H|^-1. However, since D^E is a function of the Hession which is O(n^2) in the number of weights of the network, both computing it and storing it becomes computationally intractable. Their main *practical* contribution is an efficient approximate algorithm "Equilibrated Stochastic Gradient Descent" (ESGD) which estimates the optimal D^E by using the R-operator and requiring roughly two additional gradient calculations (further amortized by only performing this step every ~20 iterations) and sampling a vector from a Gaussian.The theoretical and practical contributions are verified by their empirical results which show that ESGD outperforms both standard SGD, Jacobi preconditioning and RMSProp for training deep auto-encoder networks.Overall, the paper is fairly clearly written, although I had some issues with ambiguous notation, primarily between distinguishing between element-wise operators and matrix-operators and at some points where it appears that matrices are defined element-wise but equated with full matrices, or vice versa, e.g.:* Eqn 7 and the definition for D^-1 = 1 / ||A_{i, .}||_2 just above it. Do you mean here the i'th element of D^-1 is defined like that? Otherwise, what does the subscript i refer to in A_{i,.} ? * Eqn 10 (which I'm taking to mean that each element i on the diagonal of D^E is defined as the norm of the i'th row of the Hessian, ||H_{i,.}||, is that correct?). Overall it would be helpful to add a section at the beginning on your notation, and specifying which operators act element-wise (e.g. |A|) and how you index into matrices, e.g. what do you mean by * A_{i,.} vs A_i (I'm presuming one indexes a row from a full matrix and the other an element from a diagonal matrix?), or* q_i q_j (e.g. below Eqn 11) vs q_{j,i} (in the supplementary material). Are these both the same?In the supplementary material, it was not clear to me in Proposition 3 how the q on the RHS of the inequality is still squared after applying (q_*^2)^-{1/2} which seems to be q_*^{-1}?


Cross-Domain Matching for Bag-of-Words Data via Kernel Embeddings of Latent Distributions
We propose a kernel-based method for finding matching between instances across different domains, such as multilingual documents and images with annotations. Each instance is assumed to be represented as a multiset of features, e.g., a bag-ofwords representation for documents. The major difficulty in finding cross-domain relationships is that the similarity between instances in different domains cannot be directly measured. To overcome this difficulty, the proposed method embeds all the features of different domains in a shared latent space, and regards each instance as a distribution of its own features in the shared latent space. To represent the distributions efficiently and nonparametrically, we employ the framework of the kernel embeddings of distributions. The embedding is estimated so as to minimize the difference between distributions of paired instances while keeping unpaired instances apart. In our experiments, we show that the proposed method can achieve high performance on finding correspondence between multi-lingual Wikipedia articles, between documents and tags, and between images and tags.

The authors propose a kernel-based algorithm for cross domain matching task where the goal is to retrieve a matching instance in the target set given an instance in the source set. Each instance of the two different domains is treated as a bag of features (e.g., for words) and is embedded to the common space (RKHS) with kernel embedding for matching purpose. The features of the two domains are learned jointly by maximizing a likelihood which is (intuitively) inversely proportional to the distance (i.e., MMD) of the embedded instances in RKHS. With real data, the proposed method outperforms standard approaches like CCA and kernel CCA on document-document, document-tag, and tag-image matching tasks.The writing is clear and easy to follow. The experimental results are impressive as the method consistently outperforms all other methods in all the three experiments. In machine learning (excluding information retrieval), I understand that cross domain matching is mainly tackled by CCA or kernel CCA. So the paper provides a starting point for further exploration in this direction. Regarding the originality, the idea of embedding a bag of hidden features with kernel and learn the features is, however, not new as this was considered in [18] (Latent Support Measure Machines for Bag-of-Words Data Classification, NIPS 2014). The part which is original seems to be the use of such approach for cross-domain matching with a proposed probabilistic model similar to the one in kernel logistic regression. Although the experimental results provide an evidence that the method performs better than KCCA on certain tasks, the paper does not give enough motivation, justification and description of the advantages of the method for better understanding. My comments/questions in descending order of importance.1. From lines 70-78, the problem of kernel CCA is not clearly stated. This is an important motivating paragraph in the paper.2. Presumably the learned latent vectors x_f, y_g play more role in the algorithm than the kernel k. How does the Gaussian kernel k compare to a linear kernel with a large latent dimension q ? I have a feeling that with large q, the learned latent vectors will have enough degree of freedom to perform the task even with linear kernel. This is to say that, if my intuition is correct, a kernel is not needed. 3. What is the motivation for the likelihood in Eq. 8 ? To me, this appears to be an ad-hoc choice. What is wrong with using just ||m(X)-m(Y)||^2 as the loss in Eq. 11 ? Since the log term in Eq. 11 will disappear, this should make optimization simpler and cheaper.4. How does the method compare qualitatively (not numerically) with kernel CCA ? I suggest that the authors shorten Sec. 4.1 and Sec. 4.3 and explain it. 5. It would be interesting to see the effect of varying q (latent dimension) on the precision and run time. The experiments only focus on one aspect which is the precision.6. What is the computational cost of the method as compared to kernel CCA ? It seems kernel CCA can be more efficient. The gradient in Eq. 12 must be expensive and the objective is highly non-convex. This should be addressed in the paper.7. Line 284-286: Are the hyper-parameters chosen by cross validation ? By "development data", is it the same as a validation set ?8. In the experiments, does KCCA use the same learned featured x_f, y_g from the proposed method ? If not, what is the kernel of KCCA i.e., Gaussian kernel on what ? What is the result if you do so ?9. In abstract, lines 27-28, "..while keeping unpaired instances apart." How is this criterion implemented in the method ?10. Lines 100-102, "..can learn a more complex representation..." How ? If possible, can you show this experimentally ?Minor:1. Eq. 3: ||m(X)-m(Y)||^2 is not a distance; ||m(X)-m(Y)|| (or MMD) is.2. Should mention that x_f is a latent vector for word (feature) f in the experiments.3. In the references, the format is not consistent. Some entries are written with abbreviated author names.===== after rebuttal ======= I have considered the authors' rebuttal which answered only some of my questions.


Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.

This paper proposes a combination of deep learning and stochastic optimal control for control of non-linear dynamical systems from image inputs. The system learns a latent space representation of the true system state based on images of the state using deep variational auto encoders. Additionally, a locally-linear dynamics model is learned on the latent space which is used to generate state space trajectories by applying stochastic optimal control techniques directly on the latent space. The system is tested on four control tasks and the performance is compared with multiple baselines showing good performance.Quality: --------- The paper is logical and sound. The use of (deep) variational encoders to learn the latent space representation mirrors previous work. Enforcing local-linearity on the transition dynamics coupled with the constraint between the transition and inference distributions is crucial for good performance. It would be interesting to see if the authors can model longer term dependencies using RNNs and/or LSTMs. The experiments highlight the strength of the approach, especially the fact that the learned representations are conducive for control. More experiments on higher dimensional (state/control dimensionality) tasks would help. The authors could compare their learned latent representations with those learned by Predictive State Representation based methods.Clarity: --------- The paper is very well written and is easily understandable. The fact that the authors give details on the structure of their networks and various hyper-parameters in the supplementary materials is appreciated. In Sec 2.2, line 103, correct c_T(x_t,u_t) = c(x_T, u_T).Originality: ------------- The main novelty of the paper lies in controlling the system directly on the latent space and the restrictions on the latent transition dynamics. This makes the method more stable and improves performance significantly as compared to prior approaches.  Significance: --------------- The paper advances the state of the art in pixel based control of non-linear dynamical systems and is very relevant to the deep learning, RL and representation learning community. Related citation to add: Boots et al., Closing the Learning-Planning Loop with Predictive State Representations, RSS 2010 -> The authors learn a model of the environment from camera data and plan on the learned modelI vote for a clear accept.


Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. To fill the gaps in theory and provide theoretical supports, this paper studies two asynchronous parallel implementations of SG: one is over a computer network and the other is on a shared memory system. We establish an ergodic convergence rate O(1/ √ K) for both algorithms and prove that the linear speedup is achievable if the number of workers is bounded by √ K (K is the total number of iterations). Our results generalize and improve existing analysis for convex minimization.

The authors extended the stochastic gradient and coordinate descent to the asynchronous and nonconvex differentiable setting, assuming that the asynchronism is bounded. The authors showed that the average of the expected gradient converges to 0 at the usual sublinear rate O(1/sqrt{K}), where K is the number of iterations. The proofs appear to be solid and the authors did a fair job in comparing with relevant works in the literature. Very limited experiments were given in the appendix for a quick sanity check. Overall, the results in this work may be a good addition to the field.Quality: The proofs mostly appear to be correct, and the model, much built on previous work, is useful and relevant in practice. However, I am not quite convinced by the measure of convergence: the average of the expected gradients converge to 0 seems to be too weak. First, what can we conclude from the expectation here? If we run the algorithm for say 10 repetitions, we cannot really average the iterates to get a single iterate that has a strong guarantee. This is the cost of not having convexity. Second, the obtained results do not cover the known facts for convex functions. It would be much more convincing had one been able to include the special convex case. Some technical issues:1). In assumption 3, i_k is allowed to be dependent on xi_k. This seems to contradict Eq. (33) in line 937, i.e., conditioned on xi_k, i_k may not be uniformly distributed. This may invalidate the claim in Eq. (16) (line 406). 2). Line 814, here one should first condition on all xi_s, s < j'' (for fixed m), and then take expectation wrt xi_{j''} to zero out the term involving xi_{j''}. This is because x_{j''-tau_{j'',m}} may depend on xi_{j'} so we cannot first take expectation wrt xi_{j'}. (Also, here \nabla_{i_{j'}} should be \nabla, multiple times.) 3). Some calculations from line 1228 to 1236 seems to be messed up, but it does not seem to affect the claim. 4). The conditions in theorem 1 and 2 both involve f(x^*). This is undesirable. It would be great if one can come up with practical, usable bounds.Clarity: The paper is mostly easy to follow.Originality: The authors mostly follow the existing model in the literature but extend them to the nonconvex setting. The proofs do not involve any new techniques. So overall the work is incremental, but I think it is still of some interest.Significance: I feel that the authors oversell their work a bit, for the obtained results are far from "explaining deep learning or nonconvex stochastic optimization." The obtained sublinear rates also need more careful interpretation.Reference: The following work also seems to be relevant and perhaps should be cited: Ho et al., More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server. NIPS 2013.


Individual Planning in Infinite-Horizon Multiagent Settings: Inference, Structure and Scalability
This paper provides the first formalization of self-interested planning in multiagent settings using expectation-maximization (EM). Our formalization in the context of infinite-horizon and finitely-nested interactive POMDPs (I-POMDP) is distinct from EM formulations for POMDPs and cooperative multiagent planning frameworks. We exploit the graphical model structure specific to I-POMDPs, and present a new approach based on block-coordinate descent for further speed up. Forward filtering-backward sampling -a combination of exact filtering with sampling -is explored to exploit problem structure.

This paper presents an EM method for solving interactive POMDPs (I-POMDPS), which exploits problem structure in the I-POMDP model. Specifically, an EM method for I-POMDPs is introduced, along with improvements which use block-coordinate descent and forward filtering-backward sampling. Experimental results show significant scalability gains using some of these methods. To the best of my knowledge, this is the first EM method applied to I-POMDPs. While I-POMDPs have many similarities to POMDP (and Dec-POMDPs), where EM has been used, there is additional structure in I-POMDPs in the form of models of the other agents in the problem. As such, while EM could be applied naively to I-POMDPs, more specialized methods could also be developed. Since EM methods have been shown to perform well in problems (particularly when there is significant structure), using EM to solve the very difficult (but structured) I-POMDP problems could be promising. The proposed EM method exploits the idea that a distribution over other agent actions is sufficient instead of using a distribution over models. This allows more efficient inference, but will still have scalability issues. As a result, the authors include block-coordinate descent, which is an optimization scheme that groups sets of variables and iteratively improves them. Block-coordinate descent could be used in any EM method, but the structure of the I-POMDP may be helpful to setting the blocks.The authors also introduce a forward filtering-backward sampling (FFBS) method to improve scalability. The experimental results show significantly improved performance compared to the previous method (bounded policy iteration or BPI) --- larger problems can be solved and other problems can be solved more quickly. Nevertheless, the experiments are missing some combinations of methods making it difficult to analyze the methods. Really, the authors should compare all algorithm combinations on all domains (or give a good reason why this is not possible). The authors also chose the quickest combination of methods *for each problem* to compare against BPI. This is unfair, as you wouldn't know this a priori. Also, FFBS does not perform very well, so the authors should consider additional analysis on this method or removing those results. Additional analysis concerning the experimental results would be helpful. For instance, both BPI and EM can get stuck in local optima, but BPI has the ability to escape some local optima by adding nodes. Why is it that the EM method is able to so significantly outperform BPI? Also, it is not clear how to set the blocks in block-coordinate descent or how to set the sizes for the other agent finite-state controllers. These are key features affecting the complexity and performance of the methods, so they should be discussed in more detail. The writing is understandable, but there are several typos and grammatical errors. Additional detail should be given in section 4 (computational complexity) concerning the complexity of the methods themselves. And additional EM-based methods for POMDPs should be discussed such as: H. Li, X. Liao, and L. Carin. Multi-task reinforcement learning in partially observable stochastic environments. Journal of Machine Learning Research, 10:1131-1186, 2009.


Convolutional Neural Networks with Intra-layer Recurrent Connections for Scene Labeling
Scene labeling is a challenging computer vision task. It requires the use of both local discriminative features and global context information. We adopt a deep recurrent convolutional neural network (RCNN) for this task, which is originally proposed for object recognition. Different from traditional convolutional neural networks (CNN), this model has intra-layer recurrent connections in the convolutional layers. Therefore each convolutional layer becomes a two-dimensional recurrent neural network. The units receive constant feed-forward inputs from the previous layer and recurrent inputs from their neighborhoods. While recurrent iterations proceed, the region of context captured by each unit expands. In this way, feature extraction and context modulation are seamlessly integrated, which is different from typical methods that entail separate modules for the two steps. To further utilize the context, a multi-scale RCNN is proposed. Over two benchmark datasets, Standford Background and Sift Flow, the model outperforms many state-of-the-art models in accuracy and efficiency.

This paper presents a recurrent convolutional neural network for semantic image segmentation to encode and take advantage of contextual relationships. The method is basically a combination of [13] where a very similar RCNN is used for object recognition, and [4] from which the multi-scale pipeline is inspired. Thus, technically, the paper is not very novel (Sec. 3.1. is much the same as in previous works - maybe that should be stated more clearly; the combination can be seen as a novelty of course). However, the paper is well executed, very easy and clear to read, largely well written and provides a seemingly fair evaluation to state-of-the-art. Some recent works or interesting evaluations could be added, see below. Results are shown on the Sift Flow and the Stanford Background dataset where the proposed technique outperforms state-of-the-art using a limited number of training data for all approaches (limited by these datasets). Also, parameter ablation studies are conducted to some extend. The method is very efficient.Detailed comments:- The related work section could be written more sharply. It is not always crystal clear what the differences are with respect to the closest related works.- Results on PASCAL VOC could be added as most recent segmentation works evaluate on those datasets.- l.276 is very vague and should be made more clear.- Table 1: Why are only \gamma \in {0,1} considered? The paper could provide experiments/plots with varying gammas.- page 7: The ordering of text/plots/tables should be changed to have less interleaving text/plots/tables.- The model mentioned in l.387 should be added to the table, maybe with a footnote that it uses different training data or a line break.- Some further qualitative analysis would be nice if it fits.- This recent work seems to be missing (the first one provides even stronger results on Stanford Background): Feedforward semantic segmentation with zoom-out features Mohammadreza Mostajabi, Payman Yadollahpour and Gregory Shakhnarovich Toyota Technological Institute at Chicago@inproceedings{crfasrnn_arXiv2015,author = {Shuai Zheng and Sadeep Jayasumana and Bernardino Romera-Paredes and Vibhav Vineet and Zhizhong Su and Dalong Du and Chang Huang and Philip Torr},title = {Conditional Random Fields as Recurrent Neural Networks},booktitle = {arXiv:1502.03240}, year = {2015}}


Robust Feature-Sample Linear Discriminant Analysis for Brain Disorders Diagnosis
A wide spectrum of discriminative methods is increasingly used in diverse applications for classification or regression tasks. However, many existing discriminative methods assume that the input data is nearly noise-free, which limits their applications to solve real-world problems. Particularly for disease diagnosis, the data acquired by the neuroimaging devices are always prone to different sources of noise. Robust discriminative models are somewhat scarce and only a few attempts have been made to make them robust against noise or outliers. These methods focus on detecting either the sample-outliers or feature-noises. Moreover, they usually use unsupervised de-noising procedures, or separately de-noise the training and the testing data. All these factors may induce biases in the learning process, and thus limit its performance. In this paper, we propose a classification method based on the least-squares formulation of linear discriminant analysis, which simultaneously detects the sample-outliers and feature-noises. The proposed method operates under a semi-supervised setting, in which both labeled training and unlabeled testing data are incorporated to form the intrinsic geometry of the sample space. Therefore, the violating samples or feature values are identified as sample-outliers or feature-noises, respectively. We test our algorithm on one synthetic and two brain neurodegenerative databases (particularly for Parkinson's disease and Alzheimer's disease). The results demonstrate that our method outperforms all baseline and state-of-the-art methods, in terms of both accuracy and the area under the ROC curve.

This work was vaguely interesting as it proposed handling noisy features and labels within a semi-supervised pattern recognition framework.In medicine, a proportion of labels are often wrong (eg a diagnosis of Alzheimer's disease is often incorrect).In terms of originality, there are many papers on the subject of improving classification accuracy using the ADNI (Alzheimer's disease, AD) dataset, although not as many using the PPMI (Parkinson's disease, PD) database.It would perhaps have been interesting to see if the proposed method generalises to other sorts of data beyond diagnoses using MRI.It could also be interesting to assess the behaviour of the approach if some of the training data are deliberately mislabelled.This could be future work though.This manuscript builds on the Huang et al paper on Robust regression, which I have not read. The approach involves various settings, which were determined via a nested cross-validation procedure.This should mean that the estimated classification accuracies are relatively unbiased.For the PD dataset, there were 56 normal controls and 56 patients.Classification accuracy was 84.1%, which was higher than for the other methods.The dataset was relatively small, so it's difficult to say whether this was a statistical fluke. For the AD dataset, the proposed method also appeared to perform best, although the accuracy was not so much better than for the other methods.This could easily have been down to chance.Accuracies are not really high enough for clinical applications, but this may be down to incorrect diagnoses of patients involved.If the method had been probabilistic, then prediction could perhaps be combined with other sources of information, but this is not the case here.There is not even any mention of sensitivity and specificity, which could be used to adjust for the fact that in the real world, the prior probability of having a disease is not 0.5 (or whatever the proportions are in the training data).The manuscript uses the ADNI dataset, but does not appear to comply with their data use agreement.I do not personally approve of the requirements imposed by ADNI, and I do not know if NIPS has an editorial policy on this sort of thing.I do not see how equation 1 gets around the N > d problem, as there is no regularization involved.I do not understand why the authors propose the use of k-NN to determine class labels after having performed LDA.Figure 1 is not needed and does not add anything to the work.In the "Contributions", the authors say they "build a robust discriminative classification model".My understanding of LDA is that it is fundamentally generative, which models two Gaussian distributions that share their covariance matrices.The framework here seems to be discriminative though, treating classification as a regression problem.In equations 2 and 3, is the 1 superscript needed in $|| H (Y_{tr} - \beta X_{tr}) ||_1^1$ ?On page 4, the paper says that the first term of equation 3 contains a quadratic optimization step.This does not make sense, as the equation does not contain steps.The first term is an l_1 norm. The step from equation 3 to equation 5 needs some more rigorous justification.It currently seems like an ad hoc procedure.On page 8, the manuscript says "The magnitude of the coefficients would show the significance of contribution for that specific feature".This is not a statistical significance, and it is not clinical significance.Perhaps refer to the "relevance", rather than "significance".


b-bit Marginal Regression
We consider the problem of sparse signal recovery from m linear measurements quantized to b bits. b-bit Marginal Regression is proposed as recovery algorithm. We study the question of choosing b in the setting of a given budget of bits B = m · b and derive a single easy-to-compute expression characterizing the trade-off between m and b. The choice b = 1 turns out to be optimal for estimating the unit vector corresponding to the signal for any level of additive Gaussian noise before quantization as well as for adversarial noise. For b ≥ 2, we show that Lloyd-Max quantization constitutes an optimal quantization scheme and that the norm of the signal can be estimated consistently by maximum likelihood by extending [15].

The paper presents a method for b-bit compressive sensing, i.e., estimate x^* from y and A, where y=Q(Ax^*), Q maps a real number into a b-bit string. The algorithm uses the largest entries of A^Ty to estimate x^*/||x^*||. ||x^*|| is estimated using an MLE estimate. The main message of the paper is that two-bit compressive sensing is most efficient in terms of estimating x^* accurately. The overall message seems to be interesting and the empirical results seem to verify it. The paper is well written as well and fun to read. However, the method and the claims of this paper are not rigorous and needs some more work.a) The method just uses one step process of estimating x^* (i.e. uses A^Ty). However, such a method is known to require significantly larger sample complexity and in practice as well the method is significantly worse than the other method like IHT or L1 minimization (authors observe the same in their experiments as well).b) Due to the weak method, the error bounds for the method are also significantly worse than the existing results for compressive sensing (both standard CS and one-bit style CS). For example, the given method naturally cannot do exact recovery even when the measurements are noiseless. Moreover, the recovery is not universal that for each x^*, a new A has to be sampled. This again is not very realistic and interesting scenario to study.c) The claim that b=2-bit CS has the best sample complexity is quite confusing. Because i) the method considered itself is suboptimal (as described above) ii) the analysis of the method gives only an upper bound on samples and authors claim that "inspection of proof indicates that the dependence is tight" is a bit vague and does not imply any lower bound on the number of samples required iii) the analysis of the upper bound as well as MLE estimate of ||x||_* is also not complete and requires computing certain quantities using experiments. Overall the paper contains an interesting claim but the claim is not fully justified. Especially, the method considered to make the claim is quite weak and does not even work well in practice (in reasonably low-noise scenarios).


Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
Recently, there has been significant progress in understanding reinforcement learning in discounted infinite-horizon Markov decision processes (MDPs) by deriving tight sample complexity bounds. However, in many real-world applications, an interactive learning agent operates for a fixed or bounded period of time, for example tutoring students for exams or handling customer service requests. Such scenarios can often be better treated as episodic fixed-horizon MDPs, for which only looser bounds on the sample complexity exist. A natural notion of sample complexity in this setting is the number of episodes required to guarantee a certain performance with high probability (PAC guarantee). In this paper, we derive an upper PAC boundÕ( δ+c ) that match up to log-terms and an additional linear dependency on the number of states |S|. The lower bound is the first of its kind for this setting. Our upper bound leverages Bernstein's inequality to improve on previous bounds for episodic finitehorizon MDPs which have a time-horizon dependency of at least H 3 .

***************************************** SUMMARY ***************************************** Paper: Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning The authors tackle the problem of minimising sample-complexity for episodic fixed-horizon (undiscounted) MDPs. The main contributions are:(a) a new algorithm, inspired by UCRL\gamma for the discounted setting(b) a theoretical analysis showing that the new algorithm enjoys sample-complexity guarantees that scale optimally with the horizon, the accuracy and the confidence. The result is sub-optimal only with respect to the size of the state/space, where it scales potentially quadratically while known lower bounds are linear in the state-space.***************************************** CLARITY *****************************************Overall I found the paper relatively well executed, but it's clear the authors struggled to include as much as possible in the main paper, and the readability suffers as a consequence. I don't know how to feel about this. On the one hand it's better to include proofs in the main body (this is a theory-only paper). On the other hand, as a standalone paper it is still relatively hopeless to verify the correctness of theresults. Anyways, I read enough of the supplementary material to be convinced that the results are correct. ***************************************** QUALITY, ORIGINALITY AND SIGNIFICANCE *****************************************Things I liked:* The setting is quite a bit cleaner than the discounted setting where there are no clear demarcation points to reset the policy* The results are relatively tight. Improving the dependence on the state-space would be nice, of course, but there are many cases where H^3 is larger than |S| and so far there are no algorithms with linear dependence on |S| in this setting (presumably it is not hard to adapt MORMAX, however).* The trick to allow C > 2 in the algorithm. (for me this is the main novelty) By the way, maybe it is worth to look at [2] and see if the Good-Turing trick can (has been?) be used to solve this problem.  Things I disliked:* The proof is following more-or-less exactly as the same cumbersome proof of Lattimore & Hutter, even using the recursive substitution of the higher-moment value estimates. I wish there was a simple proof of this result and I think more time could have been taken to look for one.* The novel contribution is on the low side. The setting is different, but intuitively there is not so much difference between the discounted and episodic settings with the latter usually being a little easier. Of course it is still a lot of work to check this, but many results go through with little modification.* The paper is a bit of a struggle to read. * The algorithm (like UCRL\gamma) is less principled than it could be. Eg., samples are not used as soon as they are obtained, and samples obtained beyond a certain point are simply discarded. I understand these tricks are useful to make the proof work, but my suspicion is that they are going to make the algorithm fairly impractical. I think many of the union bounds can be replaced with uniform bounds as used in [3] (and elsewhere). Perhaps this could be investigated in the future.    Overall I am inclined to accept. I hope the authors will try (hard) to improve the readability to a slightly less specialised audience.  Minor Comments:Line 51: Perhaps mention here (as in the abstract) that the upper and lower bounds do not match on the size of the state-space Line 99: You want to say that if the set of rewards are *not* known, then you can do this transformation Line 151: "as phase" -> "as a phase"? Often: When I see "infinite horizon" I am usually thinking of undiscounted average reward setting, but here you are mostly using it the discounted setting.Line 154: URCL-\gamma -> UCRL-\gamma? Line 223: Did Kakade really exploit the low variance for MDPs? Perhaps you mean the work by Azar [1], which should probably be cited (it is using a generative model and discounting, so different, but still very related) Line 270: "current policy" -> "the current policy" Line 304: This proof sketch is basically impossible to follow without just reading the proof in the appendix. Perhaps better to give a more intuitive sketch like the previous lemma and just refer the reader to the appendix?   [1] On the Sample Complexity of Reinforcement Learning with a Generative Mode. Azar, Munos & Kappen. 2012 [2] PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs. Dietterich, Taleghan & Crowley [3] lil'UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits. Jamieson, Malloy Nowak & Bubeck