PAC Reinforcement Learning with Rich Observations
We propose and study a new model for reinforcement learning with rich observations, generalizing contextual bandits to sequential decision making. These models require an agent to take actions based on observations (features) with the goal of achieving long-term performance competitive with a large set of policies.
To avoid barriers to sample-efficient learning associated with large observation spaces and general POMDPs, we focus on problems that can be summarized by a small number of hidden states and have long-term rewards that are predictable by a reactive function class. In this setting, we design and analyze a new reinforcement learning algorithm, Least Squares Value Elimination by Exploration. We prove that the algorithm learns near optimal behavior after a number of episodes that is polynomial in all relevant parameters, logarithmic in the number of policies, and independent of the size of the observation space. Our result provides theoretical justification for reinforcement learning with function approximation.

This is a technical paper which proves PAC learnability of a complex model, but it does not make a convincing case that the model is a useful extension of standard MDPs.
The paper analyzes PAC learnability of finite-horizon Contextual MDPs. Contextual MDPs are a specific type of POMDPs with the restriction that the optimal q-function depends only on the most recent observation (instead of the belief state). The authors show that Contextual MDPs are not poly PAC learneable even when either memoryless policies are considered or value function approximation is used. However, when both memoryless policies and value function approximation is used and the transitions are deterministic, then the model is PAC learnable in a polynomial number of episodes (and the complexity is independent of the number of observations).  The paper is well written overall. The proofs are quite clear and quite thorough. I am not quite sure that the 16 pages of technical proofs in the appendix are suitable for a conference; the paper may better fit a journal format. The presentation could be further improved by better describing the results and how they were obtained earlier on in the paper. It takes a lot of effort to really understand the contribution and what are the main ideas in constructing the new algorithm.  The main limitation of the paper is that its results apply to a rather limited setting. While it is clear that Contextual MDPs are slightly more general than plain MDPs, the authors do not make a convincing case that any interesting or useful problems fall into this category. In particular the results apply to a POMDP in which 1) transitions are deterministic, 2) optimal q-function depends only on the most recent observation, and 3) the value function approximator can exactly represent the optimal q-function.  The authors only spend a single paragraph on the justification of why this may be a useful model in Example 1, but the example is quite construed.  There are very few (if any at all) plausible types of POMDPs in which the optimal q-value depends only on the last observation and action that are not also MDPs. I could imagine a model in which the optimal policy is reactive (memoryless), but the requirement of the value function to be memoryless is just too strong.   The technical result of the paper is interesting in that it proposes a sampling method that is independent of the number of observations. It is a generalization of learnability results for MDP (with the caveat that the transitions are deterministic), but the proof and algorithms are much less elegant.   Perhaps is the paper was posed in terms of learnability of MDPs with value function approximation, the  results would be more meaningful and applicable.  More detailed comments: 1) Introduction: the first few paragraphs are quite vague, the references (e.g. [21]) do not really support the claims, and the paper does not at all answer the question posed. 2) The layered process is really just a standard finite-horizon MDP (with non-stationary transitions) 3) Appendix A: Please start with the description of what is in the appendix. One needs to read the entire Theorem 2 to understand that it is just a technical lemma needed to prove Prop 1 & 2. 4) Section 2.2, Definition 1: This is a bit repetitive given the description in 2.1 5) Assumption 1: The requirement to have only deterministic transitions limits the applicability of the results quite a bit 6) Section 3.1: The algorithms and basic ideas of the proof are described very nicely in this section. 7) References have several instances of incorrect capitalization 8) Line 404: The argument regarding no need to consider stochastic policies is not convincing. I guess the equivalence follows from “splitting policies in MDPs”, but it would be good to provide more details.


Maximizing Influence in an Ising Network: A Mean-Field Optimal Solution
Influence maximization in social networks has typically been studied in the context of contagion models and irreversible processes. In this paper, we consider an alternate model that treats individual opinions as spins in an Ising system at dynamic equilibrium. We formalize the Ising influence maximization problem, which has a natural physical interpretation as maximizing the magnetization given a budget of external magnetic field. Under the mean-field (MF) approximation, we present a gradient ascent algorithm that uses the susceptibility to efficiently calculate local maxima of the magnetization, and we develop a number of sufficient conditions for when the MF magnetization is concave and our algorithm converges to a global optimum. We apply our algorithm on random and real-world networks, demonstrating, remarkably, that the MF optimal external fields (i.e., the external fields which maximize the MF magnetization) shift from focusing on high-degree individuals at high temperatures to focusing on low-degree individuals at low temperatures. We also establish a number of novel results about the structure of steady-states in the ferromagnetic MF Ising model on general graph topologies, which are of independent interest.

The paper proposes an Ising network based model for opinion dynamics and proposes the Ising influence maximization method based on it. Under mean field approximation, it develops several sufficient conditions for when the problem is convex and solvable and a projected gradient descent algorithm. It also establishes several results about the structure of steady-states in the ferromagnetic MF Ising model. It conducts experiments on both synthetic and real world authorship datasets.
The paper studies the influence maximization using the Ising network model. However, the motivation for using Ising network model for opinion dynamics is not well motivated. Why do we need to use the Ising model? It is less convincing by just saying that this model can capture the steady-state of the opinion dynamics. Some other model can also consider the steady state. For example, how does it compare with prior work such as [1]. In [1], it also considers modeling the steady state of the opinion dynamics. Moreover, how does the discrete-time Ising compare with the continuous-time model in [1]? Some prior work is missing and the paper should have a paragraph clearly stating the stand of this method with respect to other work.  As for experiments, it would be good to compare with other standard baselines in terms of the influence maximization. In the experiments it only compares with three heuristics, which is less convincing. Moreover, it would be good to add the scalability experiments on large networks.  As for writing, proof sketch could be moved to the appendix. Moreover, the theorems stating sufficient conditions for existence of steady state are less connected to influence maximization, hence makes section 4.1 less connected to other sections. Some terms are also not well defined. For example, what does the susceptibility mean in equation (8)? Any citation on how to obtain equation (5)?   Several important literatures are missing: [1]. Learning Opinion Dynamics in Social Networks [2]. Influence maximization in continuous time diffusion networks [3]. Scalable Influence Estimation in Continuous-Time Diffusion Networks [4]. Scalable influence maximization in social networks under the linear threshold model [5]. A data-based approach to social influence maximization


Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
Offline handwriting recognition systems require cropped text line images for both training and recognition. On the one hand, the annotation of position and transcript at line level is costly to obtain. On the other hand, automatic line segmentation algorithms are prone to errors, compromising the subsequent recognition. In this paper, we propose a modification of the popular and efficient Multi-Dimensional Long Short-Term Memory Recurrent Neural Networks (MDLSTM-RNNs) to enable end-to-end processing of handwritten paragraphs. More particularly, we replace the collapse layer transforming the two-dimensional representation into a sequence of predictions by a recurrent version which can select one line at a time. In the proposed model, a neural network performs a kind of implicit line segmentation by computing attention weights on the image representation. The experiments on paragraphs of Rimes and IAM databases yield results that are competitive with those of networks trained at line level, and constitute a significant step towards end-to-end transcription of full documents.

The authors investigate the introduction of an attention mechanism into an offline handwriting recognition system pointed at unsegmented paragraphs of handwritten text. MD-LSTMs are used for feature extraction, attention computation, and (in some cases) decoding. Interestingly, learned attention is shown to outperform ground truth segmentation in the authors' system. Character error rates on standard datasets appear to be quite impressive. While the system's word error rates do not break state of the art, they are quite competitive despite ignoring both ground truth segmentation and employing a relatively simplistic language model.
Positives: -The idea of applying technique to generalize single-line handwriting recognition to multi-line is both an elegant idea and a natural next step. It is unsurprising that this is one of at least two works (mentioned by the authors) attempting similar techniques (not that this is relevant, but I consider this to be the more mature/developed work of the two). -It is impressive (as is often the case these days) that such a system works at all, let alone that it works as well as it does. -The technical exposition provided by the authors is sufficient, and is generally easy to follow. The colorized attention figure is particularly gratifying.  Areas for improvement: -The comparison to established benchmarks raises more questions than it provides answers. As the authors note, this comparison is apples-to-oranges. The reference models exploit more complex language models and make use of ground truth segmentation. The latter is unavoidable due to the fundamental nature of the problem considered, but the gap in language models is quite unnecessary. Discussion of this comparison also feels underdeveloped and unenlightening, and I am confident the authors could do better. -The authors do not adequately discuss the significance/origin of the learned attention outperforming the ground truth. Is it because of noise in the annotations? Leveraging information contained in neighboring lines? This is particularly important, as the models compared to in Sec 5.4 all make use of this ground truth. -Finally, while the paper is well organized and easy to follow, the quality of the writing leaves something to be desired. There are far too many typos, grammatical errors, and shifting tenses. Furthermore, particularly when comparing against other work, the occasionally competitive tone undermines the authors' credibility. Their results stand up quite well, even in a neutral light.


CRF-CNN: Modeling Structured Information in Human Pose Estimation
Deep convolutional neural networks (CNN) have achieved great success. On the other hand, modeling structural information has been proved critical in many vision problems. It is of great interest to integrate them effectively. In a classical neural network, there is no message passing between neurons in the same layer. In this paper, we propose a CRF-CNN framework which can simultaneously model structural information in both output and hidden feature layers in a probabilistic way, and it is applied to human pose estimation. A message passing scheme is proposed, so that in various layers each body joint receives messages from all the others in an efficient way. Such message passing can be implemented with convolution between features maps in the same layer, and it is also integrated with feedforward propagation in neural networks. Finally, a neural network implementation of endto-end learning CRF-CNN is provided. Its effectiveness is demonstrated through experiments on two benchmark datasets.

This paper explores a CRF-CNN architecture for human pose estimation. The paper presents a an exposition of how to construct a CRF-CNN model and pass messages. Strong experimental results are provided.
The general idea of this work is clearly in a direction of interest to the community and the results look strong. However, there are a few aspects to this work that I find quite unclear.  If they were clearer this work would have much more potential for impact.  In particular, it is not clear enough if this work uses a truly 'end-to-end' approach (as stated for one of the contributions).  For example on line 46 it is stated that: "We show step by step how approximations are made to use an end-to-end learning CNN for implementing such CRF model."  On line 114 it is stated that "Therefore, the mapping from f to h ̃ can be implemented with one-layer transformation in a neural network and sigmoid is the activation function.  h ̃ is a new feature vector derived from f and f can be obtained from lower layers in a network."  However, there is no discussion about the procedure or the possibility of back propagating information into the CNN. It is stated on line 151 that "We use the fully convolutional implementation of VGG and the output of fc6 in VGG is used as the feature map".  It is therefore unclear precisely what is being done with respect to this issue.  * Is the CNN component being trained here through backpropagation, or is this a CRF on top of CNN features?  There has been a fair amount of work that has looked at these types of random field models for human pose prior to the explosion of interest in CNNs. See the CRF work of:  Kiefel, M. and Gehler, P.V., 2014, September. Human pose estimation with fields of parts. In European Conference on Computer Vision (pp. 331-346). Springer International Publishing.  and the references therein for pointers to some of the other prior work. I agree that while there are a number of examples of prior work that are very close to this formulation, there is a need for work that combines CNNs and CRFs in clear and clean ways.  The work presented here looks a lot like it boils down to taking features of a CNN as input into fairly well known CRF models, and therefore why not just use established inference methods like sum/max product for trees and mean field for loopy graphs  ?  It is also not clearly and explicitly stated why an inference procedure consisting of 'standard' sum/max product for a trees or mean-field for loopy graphs is or is not being used or compared with here. The motivation and justification of the proposed messages is unclear. These messages should be much more explicitly compared and contrasted with the corresponding established message passing equations. Ideally this comparison would be made both conceptually and empirically.  The abstract asserts: "A message passing scheme is proposed, so that in various layers each body joint receives messages from all the others in an efficient way".  While the manuscript is fairly clear and detailed regarding the CRF inference equations, these equations are not clearly motivated and related to the well known inference procedures of loopy BP and mean field.  On line 109 it is stated that "p(h|I , Θ) in (11) is approximated by a product of independent Q(hi |I , Θ) in (13) and (14), similar to mean-field approximation."  Why not use mean field exactly? In the case of loopy graphs, does the proposed algorithm become loopy BP? Since some of the motivation presented here is related to approximating h~ with an expectation (ex. (15)), is there a connection between the proposed procedure and expectation propagation?  The results regarding 'flooding' vs serial updates in Table 3 seem like they should be somewhat unsurprising if viewed through the lens of formally derived mean field updates. i.e. According to mathematics of mean field, updates should be done serially - each local Q is updated based on the current Q of neighbours. If viewed through the lens of loopy BP, loopy BP has been well studied and a more detailed placement of the procedure here with known results from loopy inference should be provided.  I think the paper would benefit greatly from comparing and contrasting both theoretically and empirically the proposed scheme with established mean field and loopy BP update methods.   The general idea of merging CRFs and CNNs is certainly of current interest and the quality of the exposition and exploration here will have an important influence on the long term potential for this paper to have impact.  The author response on these issues could alter my numerical ratings above.


Fairness in Learning: Classic and Contextual Bandits *
We introduce the study of fairness in multi-armed bandit problems. Our fairness definition demands that, given a pool of applicants, a worse applicant is never favored over a better one, despite a learning algorithm's uncertainty over the true payoffs. In the classic stochastic bandits problem we provide a provably fair algorithm based on "chained" confidence intervals, and prove a cumulative regret bound with a cubic dependence on the number of arms. We further show that any fair algorithm must have such a dependence, providing a strong separation between fair and unfair learning that extends to the general contextual case. In the general contextual case, we prove a tight connection between fairness and the KWIK (Knows What It Knows) learning model: a KWIK algorithm for a class of functions can be transformed into a provably fair contextual bandit algorithm and vice versa. This tight connection allows us to provide a provably fair algorithm for the linear contextual bandit problem with a polynomial dependence on the dimension, and to show (for a different class of functions) a worst-case exponential gap in regret between fair and non-fair learning algorithms.

This paper also studied the fairness in learning algorithm, along with another submission paper ID 212 which I also reviewed.  Paper ID 212 studied the case with classical multi-armed bandit. This paper considered the more general contextual bandit case, where a contextual bandit algorithm is considered \delta-fair, if it never selects an option with probability strictly greater than another better option.  The paper establish an equivalence between fair contextual multi-armed bandit and the KWIK learning. Roughly speaking, if a function class C is KWIK learnable, then the corresponding fair contextual multi-armed bandit problem can achieves a non-trivial regret after a polynomial number of rounds; and if the fair contextual multi-armed bandit problem admits a non-trivial algorithm, then the function class C is KWIK learnable. The proved guarantee is rather complicated, and indeed establish that these two concept are polynomially related.   In order to show that, the paper first gives a simple algorithm for the fair contextual multi-armed bandit problem. The idea is very simple, it associates a learner with carefully chosen parameters for each arm, and then it use the learners to provide a confidence range for each arm. In order to be fair, the algorithm treat all arms with overlapping confidence range as equal, and thus crate an equivalence class between the arms. Then it pick a random arm within the equivalent class containing the arm with highest upper range, and feed a sample to that.  There are some technical points like rolling back the history for the learner in order to deal with the bandit feedback setting, but overall the algorithm is intuitive and elegant.  Then the paper gives an algorithm for KWIT using a fair contextual multi-armed bandit problem as a subroutine. The high-level idea is that intuitively, since the algorithm is fair, it can only exploit if it is quite certain about the hidden function f. The reduction is also straightforward and the analysis is elegant.
Technical quality:  I am confident that the results are correct.  But there are some issues in the proof. The (b) part in Lemma 1 seems not right, it should be the total number of times that s_i^t is not I don't know and i does get sample instead of the stated one, which seems wrong to me. Also, in Lemma 1 it says arm j but it use i later.  But the above issue does not affect the validity of the later proof, as it uses the right version. But this somehow makes me misunderstand something when I first read that.  Novelty/originality:  The study of the fairness in learning seems both interesting and of practical importance to me. And their definition is clearly original.  The connection between fair contextual multi-armed bandit and KWIK learning is a fantastic result. It is also nice to know that there is an exponential separation between fair and non-fair algorithm when considering the contextual problems.  Potential impact or usefulness:  I expect that this paper along with the author's another paper 212 will influence a number of future works in addressing the fairness concern in learning: i.e., maybe their definition can be applied to settings other than contextual multi-armed bandit, or it can inspired some other definition of fairness in learning, which has a weaker impact on the regret achievable.  Clarity and presentation:  The paper is generally well-written, but there are many typos/errors in some part of the paper.  A partial list of typos/errors:  As said before, the statement for Lemma 1 is not correct and need to be fix.  In line 15 of the KWIKTOFAIR algorithm, it should be "update the history for L_{j*}"  One Line 230, the total regret should be 2k\eps^* \cdot T + n +\delta T.  On the first line on page 6, n_i seems to mean the number of rounds that arm i outputs I don't know, instead of "some arm".  There is a unnecessary ) in the equation after line 237.  On line 9 of the FAIRTOKWIK algorithm, the x in A(h^t,x^t,x) should be x(\hat{\elk}).  I feel like the submission still needs some revisions.


β-risk: a New Surrogate Risk for Learning from Weakly Labeled Data
During the past few years, the machine learning community has paid attention to developing new methods for learning from weakly labeled data. This field covers different settings like semi-supervised learning, learning with label proportions, multi-instance learning, noise-tolerant learning, etc. This paper presents a generic framework to deal with these weakly labeled scenarios. We introduce the β-risk as a generalized formulation of the standard empirical risk based on surrogate marginbased loss functions. This risk allows us to express the reliability on the labels and to derive different kinds of learning algorithms. We specifically focus on SVMs and propose a soft margin β-SVM algorithm which behaves better that the state of the art.

The paper proposes a surrogate risk for weakly supervised learning. The risk allows for a per example weighting which can be adapted to different learning scenarios. A condition is determined under which the proposed risk is a tight bound to the 01 risk. This is used to derive an iterative optimisation algorithm. Experiments are presented for semi-supervised and noisy label learning.
UPDATE AFTER REBUTTAL:  I still feel the value of the framework isn't fully convincing. My basic issue is that for the weakly supervised scenarios that don't already have principled algorithms, the precise reason the provided formulation is superior is unclear. For example in SSL the proposed method has a very similar flavour to self-training. I like however that there is an attempt at an unified approach to a range of such problems, and it's possible that one could do something interesting with this framework in future work.   On the technical side, I still don't quite get the optimisation proposed for beta (Line 144). As I understand, Eqn 4 implies   R_\phi^\beta(X, h) = R_\phi(S, h) + (1/m) sum_{i = 1}^m beta_i y_i h(x_i)   so it seems like optimising R_\phi^\beta(X, h) is equivalently optimising the weighted mean term. But as that's constrained to be zero in the optimisation, we're just finding any feasible beta? The authors comment that there is a constraint on beta that is needed for the above equality, but this constraint seems indeed imposed in the optimisation.  The basic idea of constructing new risks that can adapt to different learning scenarios is interesting, and in keeping with some recent work for special cases e.g. Patrini et al. 2013 for learning from label proportions. The paper begins with a decomposition of an arbitrary supervised risk into a weighted unsupervised risk, plus an additional term which is a certain beta-weighted mean. The idea is then that if one can somehow omit the second term, one can learn using the first term alone, which is purely unsupervised.  The precise decomposition given in Lemma 1 is certainly interesting, and appears novel, but seems a simple extension of Thm 3 of Patrini et al. 2013 (specifically, the result is implicit in the proof of said theorem), or Lemma 1 of   Patrini et al. Loss factorization, weakly supervised learning and label noise robustness. ICML 2016.  Thus, I think the key issue is whether the result is compellingly used for new weakly supervised learning problems. On this front, I think the paper perhaps falls a bit short. It starts with a nice observation, which perhaps is intended to be the key insight of the paper: to make the new beta risk and the original as close as possible, one must constraint the beta-weighted mean of the classifier to be zero. This appears a departure from previous work, and gives a concrete criterion to learn from weakly supervised data. However, this term depends on the unobserved labels for the instances, and so the proposed method unavoidably runs into the issue that purely learning from unlabelled data is impossible without further assumptions.  The chosen approach to get around this is, unsurprisingly, a heuristic: one guesses the labels for the instances based on the current model (reminiscent somewhat of self-training for semi-supervised learning). The framework would have been more compelling were there something cleverer that could be done here. Given that a heuristic is employed, I think the paper could have explained in more detail why the proposed scheme is more attractive than existing proposals (unified framework?).  With this, the paper seeks to iteratively optimise for a good classifier, and the set of per instance weights beta. The optimisation of the classifier is standard; for the beta weights, one seeks to minimise the new risk subject to the constraint that the weighted mean is zero. while the latter is intuitive, I was unsure as to the well posedness of this optimisation. In the second stage of the iterative algorithm of section 3, the object that is optimised is R(h, beta; X). As I understand, section 2.2 establishes that this is equivalent to R(h, beta; S) + mu(beta), where mu(beta) is the weighted mean that is constrained to 0. But isn't R(h, beta; S) further just R(h; S) by Lemma 2.1, in which case the objective is effectively mu(beta), as the other term is independent of the precise choice of beta? In which case we would simply pick any admissible beta which satisfied the constraint? (This is potentially ok, as it doesn't affect the final risk? But the issue seems unclear.)  The experiments presented are promising, showing the potential to improve upon the WellSVM in certain cases.  Overall, the paper has an interesting perspective on weakly supervised learning, albeit one that builds upon similar proposals for special cases in Patrini et al. The proposed method is simple, and seems to perform well in practice. However, the method perhaps falls short of being a theoretically justified, unified solution to weakly supervised learning: it unavoidably faces the problem of having to make assumptions to get around the lack of labels, and does so in an acceptable if not particularly insightful way.  Other comments:    - it might be more standard to use \ell instead of \mathcal{L} for the loss function    - consider using e^-x or exp(-x) instead of exp^{-x}    - explicitly show dot product or transpose in definition of Bregman divergence    - the discussion about seeking an upper bound to the 01 risk could perhaps comment on the further need for a regret bound between the 01 and the newly proposed risk. Otherwise, the optimisation would have no guarantee of actually converging to the correct solution.    - the notation ^t\beta for the iterates seems awkward to me.    - It is stated that one estimates labels based on the current set of weights beta. in the case of semi-supervised learning, the weights are initialised to 0.5. How does one pick weights in this case? Is it a random flip?    - kernel should be on X x X, not X^2?   - the proposed method requires that one learns a separate wait for every instance. When the number of instances is large, this might be problematic. Can you comment on possible ways of alleviating the burden?    - consider having markers for the points displayed in the plots.   - for citation [13], capitalize SVMs.


Simple and Efficient Weighted Minwise Hashing
Weighted minwise hashing (WMH) is one of the fundamental subroutine, required by many celebrated approximation algorithms, commonly adopted in industrial practice for large -scale search and learning. The resource bottleneck with WMH is the computation of multiple (typically a few hundreds to thousands) independent hashes of the data. We propose a simple rejection type sampling scheme based on a carefully designed red-green map, where we show that the number of rejected sample has exactly the same distribution as weighted minwise sampling. The running time of our method, for many practical datasets, is an order of magnitude smaller than existing methods. Experimental evaluations, on real datasets, show that for computing 500 WMH, our proposal can be 60000x faster than the Ioffe's method without losing any accuracy. Our method is also around 100x faster than approximate heuristics capitalizing on the efficient "densified" one permutation hashing schemes [26,27]. Given the simplicity of our approach and its significant advantages, we hope that it will replace existing implementations in practice.

The paper comes up with a novel method to find the Jacaard similarity by viewing elements of vectors x,y as probabilities instead of positive integers, and uses a simple (if almost obvious in retrospect) rejection sampling / hashing method to derive these probabilities. Experimental figures are shown, and discussion about the hashing method is given.
The authors achieve two things in their paper: One, improving weighted minwise hashing to compute the Jacaard similarity in time an order of magnitude less than Ioffe's method. Two, which I feel is more important, a simple yet novel method which is easily understandable and implementable.  To give a comparison, Ioffe's paper took a few days to read and absorb (which is understandable, since was then "the best" method so far), and one could imagine a new paper having more complex arguments / algorithm to beat Ioffe's results (in terms of accuracy and running time). This paper was remarkably easy to read and understand, and is obvious in retrospect.  Thus, this paper gets high scores in technical quality, novelty, and impact.  Experimental results are well presented.  Some small details: Surprisingly, I found the proof in the supplementary material for Theorem 1 harder to understand (notation is a bit confusing); an easier argument would be to argue that for the number of steps (hash function output) for two pairwise vectors to be the same ; this is simply adding up the probability that : the hashes are both zero, both one, both two , both three, both four, .....   This is an infinite geometric sum, but it factors out nicely into the result required using some high school algebra.   One note: seeds may be different across different languages / version of programming languages - so storing these seeds would not necessarily work all the time - although it would be easy to re-generate these hashes.   Furthermore, Algorithm 3 and 4, Figure 1 are a bit dense / confusing. For example, the red green map in Figure 1 could have dotted lines extending down for each M_is for easier intuition, and better spacing of x_is, y_is.   This is not to say the algorithms / figures should not be given. Rather, reading the paper gave the general idea, and it was writing out sample code / simulations in Matlab that understood what Algorithm 3 / Algorithm 4 does. Perhaps put the steps of these algorithms on the last page or in the supplementary material since the steps make the method seem much harder than it really is.


Equality of Opportunity in Supervised Learning
We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. We enourage readers to consult the more complete manuscript on the arXiv.

Fairness in prediction is a problem that is becoming more and more acute nowadays. Suppose we are looking to find some classifier Z which we want to be fair towards a "protected" attribute A (e.g., race). The naive approach is to require that given  Pr[Z=1 | A=a] is the same for all values that A take, so that from knowing the classification of a datapoint leads to no advantage in guessing the point's attribute. This paper introduces an alternative notion of fairness. Given a sensitive attribute an a true label Y, then the classifier "equalizes odds" if for any value of y we have that Pr[Z=1 | Y=y, A=a] = Pr[Z=1 | Y=y, A=a'] for any two values a, a' of A. (It "equalizes opportunity" if the above holds solely for Y=1.) Hence, knowing the prediction gives no more advantage in inferring the sensitive attribute when one also knows the outcome Y.  The paper then goes to proving that post-processing a Z into a Z' that satisfies either property is feasible in poly-time, given the contingency table of (Z, Y, A), for binary predictors and also for predictors Z that use a "score" in the range [0,1] to classify. They also bound the "price" of fairness in the base of Bayesian classification and finally they test their approach on a FICO data of credit scores and the loaners who actually defaulted on their loans, in partition to race. None-surprisingly, the equal-odds classifier does slightly poorly than the non-fair classifier, but clearly outperforms the classifiers that preserve the naive notion of fairness.
This is a highly novel approach for fairness that is likely to have a great impact on machine learning. The paper is extremely interesting, very novel, has some nice theoretical results and interesting experiment, and moreover - it leads to *MANY* follow-up questions. I thus recommend it for full oral presentation.  My only complaint regarding the paper is regarding its style and presentation. I feel that a more detailed version (in the supplementary material section) with clarifications and *better figures* would have resulted in a much clearer paper. I had hard time following the proofs (which aren't wrong, just require some non-trivial deductions and keeping track of notation), and clearly, a longer pass at the paper would have contributed greatly to a better understanding. I urge the authors to do so, as I think this paper is clearly "one for the books" and I suspect it will be studied in depth in the future.  A few more specific styling comments: * Equation 3.1: "Similarly the second component is the false negative rate" --- isn't it the true positive rate? Isn't false negative rate be the second coordinate of \gamma_a(1-\hat Y)? * Lemma 3.2 (nitpicking) use parentheses for second coordinate $(\gamma_i(\hat Y))_2$  * Lemma 3.2, below: a bit more discussion on this polytope wouldn't hurt. In particular -- why (0,0) & (1,1) are there? (They are the "demographic classifiers", but why are they required? I couldn't see it from the proof of 3.3 in A.1) If \hat Y is multi-labeled, would you put all \gamma_a(Y=y) points? * Figure 1, legend: aren't the equal opportunity points w.r.t to \tilde Y? (i.e., it's \tilde Y=0 and \tilde Y = 1?) If not, then I really have missed something at your explanation and/or in the understanding of the figure... * Figure 2 completely confused me. Its explanation is baffling, and it seems unrelated to any of the text that actually appears in Section 3.2. Consider replacing it or putting it in the appendix with *MUCH* more instructions as to how the reader should interpret it. * The curve C_a(t) --- seems like, from the figure, it is (Pr[R < t | ...], Pr[R < t | ...]), or I just confused because you drew in Section 5 the CDFs? * Equation (3.3) unclear what you mean by convex-hull - especially since Figure 2 has the entire area below the curve highlighted, but then you say in line 176 that you ignore points below the main diagonal. (Am I right in understanding they are "bad than a random guess" because such points give pts with Y=0 better probability of getting classified as positive than pts with Y=1? This too, which seems simple, required me to parse through some definitions - which would have been far easier if you left room for more discussion...) * Figure 3 contributes nothing to the discussion. I suggest removing it. * Proof of 4.3: why not run the conversion proving (2) for every Y\in C without changing the classification cost, hence the conversion also holds for the best Y? * Figure 6: hard to parse. What is "Raw" and what is "within group"? * Proof of Proposition 3.4: Why not simply write the LP itself? Moreover, the proof gives the impression you just need variables for Pr[\tilde Y=\hat Y], but without dependence on A you cannot write the constraint \gamma_0(\tilde Y)=\gamma_1(\tilde Y).  * Proof of Thm 4.5, line 395 --- it is not easy for me. I had a hard time following... This is an example where pic would be useful...


Truncated Variance Reduction: A Unified Approach to Bayesian Optimization and Level-Set Estimation
We present a new algorithm, truncated variance reduction (TRUVAR), that treats Bayesian optimization (BO) and level-set estimation (LSE) with Gaussian processes in a unified fashion. The algorithm greedily shrinks a sum of truncated variances within a set of potential maximizers (BO) or unclassified points (LSE), which is updated based on confidence bounds. TRUVAR is effective in several important settings that are typically non-trivial to incorporate into myopic algorithms, including pointwise costs and heteroscedastic noise. We provide a general theoretical guarantee for TRUVAR covering these aspects, and use it to recover and strengthen existing results on BO and LSE. Moreover, we provide a new result for a setting where one can select from a number of noise levels having associated costs. We demonstrate the effectiveness of the algorithm on both synthetic and real-world data sets.

This manuscripts introduces a new algorithm called "truncated variance reduction" (TruVaR; I will write TV here) that is appropriate both for level set estimation and Bayesian optimization on nonlinear functions. The idea is to maintain a set of points that are either possible maximizers of the function (for Bayesian optimization) or points near the level set threshold (for LSE). We then greedily evaluate the function so as to minimize the total truncated variance of points in this set. When the maximum uncertainty in this set drops below the current truncation value, we decrease that value and continue. The authors provide a theoretical guarantee for the performance of this algorithm, providing a unified analysis for both LSE and Bayesian optimization in terms of a notion they call "epsilon-accuracy." The authors also consider the cases where the observation noise is heteroskedastic and when there is a known cost of observing a given point, providing simple extensions of the algorithm for these cases and proving a result in the nonuniform cost case. Finally, a series of experiments investigates the performance of the proposed method on the LSE and Bayesian optimization tasks.
This manuscript has some interesting ideas. The algorithm is simple, easy-to-understand, and the fact that it works is intuitive. The joint analysis of Bayesian optimization and LSE via "epsilon-accuracy" is nice, although it's not terribly surprising that this is possible. The analysis is somewhat straightforward and predictable, using many tools that are now common in algorithms like these (confidence bounds, maximum mutual information, submodularity). That said, the theoretical contributions certainly strengthen the proposed method.  That said, I have a few complaints about this paper. The joint analysis of LSE and Bayesian optimization is interesting and perhaps even insightful. I think the LSE problem is interesting and deserving of more attention than it gets, which seems to at the rate of a paper every few years. Where this paper fails, however, is in purporting that these problems are of equal practical importance. This is not done so explicitly, but rather implicitly by certain choices made, e.g., in the experimental setup (see below). Although LSE is a compelling problem, and indeed has some applications, e.g., in environmental monitoring, it has a rather niche role in the scheme of things. This somewhat limits the potential impact of this paper. Bayesian optimization, on the other hand, has many natural applications across a range of domains. I say this not because Bayesian optimization is my pet problem (it's not), but rather out of simple pragmatism. A strong algorithmic contribution to Bayesian optimization can have a large impact indeed.  Unfortunately, I don't think this paper does a sufficient job of selling the proposed algorithm to that audience, due to the experimental treatment.  I believe the experiments are rather weak in comparison to the theoretical contributions here. LSE is investigated first, on an interesting real-world environmental monitoring dataset. The authors investigate the performance of TV against VAR (uncertainty sampling), a previous LSE algorithm, and the "straddle" algorithm from Bryan and Schneider, with TV appearing to work well but perhaps not quite as well as straddle. They turn their attention to the incorporation of cost (in this case, defined with respect to the movement of the autonomous vehicle), and show that TV outperforms LSE when measuring performance vs cost. This is not surprising due to the fact that LSE is agnostic of cost. The authors point out that their algorithm often causes the boat to remain stationary, instead choosing to measure at the same location but at a different depth. Even then, the boat moves around quite a bit along the transect. I can't help but ask how a simple grid design would perform, which could be realized with very little movement cost.  The authors then turn their attention to Bayesian optimization, the more important of the two problems, and in my opinion give it short shrift experimentally. The test functions are synthetic draws from Gaussian processes and a single hyperparameter tuning experiment (where the results are inconclusive). I do not judge these to be sufficient to convince the Bayesian optimization community that "yet another acquisition function" is needed in an already crowded space.  Varying costs of observation are also ignored, with the authors claiming that the LSE experiments "already demonstrated the effects of costs." I disagree. The LSE experiments show only that TV outperforms the cost-unaware LSE algorithm. Moreover, varying observation cost has received much attention in Bayesian optimization, and offering a better solution to that problem would significantly raise the impact of this contribution. Instead, the authors sweep it under the rug. A comparison against, e.g., EI / cost would go a long way here.  I do not understand the sentence "Since the domain is discrete, we compare only against EI and GP-UCB." Entropy search is certainly tractable here, if only via sampling and rank-one updates.  Is 24 * 14 * 4 (= 1344) a typo? 25 * 14 * 4 (= 1400)?  Finally, the experiments are riddled with magic constants, some of which are not really motivated (although citations are given for some). For example, why the exact set of costs {15, 10, 2}? Why not some other set? Reducing the noise variance from 0.05 to 1e-6 is a 223-fold increase in fidelity. Why is that only 7.5 times more expensive?  The manuscript is relatively well written and easy to read. However, many of the figures are difficult to read due to tiny text and/or small markers/details (e.g. Figure 1, Figures 2(e) and 2(f)). Reading the paper, I wondered whether there might be a nontrivial connection between TV and entropy search. It seems like TV will by construction be reducing the entropy of p(x* | D), if only indirectly. A bit of discussion on that might be interesting. For example, could it be possible that TV is approximately performing entropy search, using a cheap surrogate? If so, is it possible this could be made precise?


Improved Error Bounds for Tree Representations of Metric Spaces
Estimating optimal phylogenetic trees or hierarchical clustering trees from metric data is an important problem in evolutionary biology and data analysis. Intuitively, the goodness-of-fit of a metric space to a tree depends on its inherent treeness, as well as other metric properties such as intrinsic dimension. Existing algorithms for embedding metric spaces into tree metrics provide distortion bounds depending on cardinality. Because cardinality is a simple property of any set, we argue that such bounds do not fully capture the rich structure endowed by the metric. We consider an embedding of a metric space into a tree proposed by Gromov. By proving a stability result, we obtain an improved additive distortion bound depending only on the hyperbolicity and doubling dimension of the metric. We observe that Gromov's method is dual to the well-known single linkage hierarchical clustering (SLHC) method. By means of this duality, we are able to transport our results to the setting of SLHC, where such additive distortion bounds were previously unknown.
that of finding an optimal ultrametric tree, can be solved in polynomial time [11]. An ultrametric tree is a rooted tree where every point is equidistant from the root-for example, ultrametric trees are the outputs of hierarchical clustering (HC) methods that show groupings in data across different resolutions. A known connection between HC and MDS is that the output ultrametric of single linkage hierarchical clustering (SLHC) is a 2-approximation to the optimal ultrametric tree embedding [16], thus providing a partial answer to the NTP. However, it appears that the existing line of work regarding NTP does not address the question of quantifying the ∞ distance between a metric (X, d X ) and its optimal tree metric, or even the optimal ultrametric. More specifically, we can ask: Question 1. Given a set X, a metric d X , and an optimal tree metric t opt X (or an optimal ultrametric u opt X ), can one find a nontrivial upper bound on d X − t opt X ∞ (X×X) (resp. d X − u opt X ∞ (X×X) ) depending on properties of the metric d X ?
The question of distortion bounds is treated from a different perspective in the discrete algorithms literature. In this domain, tree embeddings are typically described with multiplicative distortion bounds (described in §2) depending on the cardinality of the underlying metric space, along with (typically) pathological counterexamples showing that these bounds are tight [4,10]. We remark immediately that (1) multiplicative distortion is distinct from the additive distortion encountered in the NTP, and (2) these embeddings are rarely used in machine learning, where HC and MDS methods are the main workhorses. Moreover, such multiplicative distortion bounds do not take two considerations into account: (1) the ubiquitousness of very large data sets means that a bound dependent on cardinality is not desirable, and (2) "nice" properties such as low intrinsic dimensionality or treeness of real-world datasets are not exploited in cardinality bounds.
We prove novel additive distortion bounds for two methods of tree embeddings: one into general trees, and one into ultrametric trees. These additive distortion bounds take into account (1) whether the data is treelike, and (2) whether the data has low doubling dimension, which is a measure of its intrinsic dimension. Thus we prove an answer to Question 1 above, namely, that the approximation error made by an optimal tree metric (or optimal ultrametric) can be bounded nontrivially.
To see this, observe that any ultrametric is a tree, and that SLHC yields an ultrametric u X that is bounded above by d X .
An overview of our approach. A common measure of treeness is Gromov's δ-hyperbolicity, which is a local condition on 4-point subsets of a metric space. Hyperbolicity has been shown to be a useful statistic for evaluating the quality of trees in [7]. The starting point for our work is a method used by Gromov to embed metric spaces into trees, which we call Gromov's embedding [12]. A known result, which we call Gromov's embedding theorem, is that if every 4-point subset of an n-point metric space is δ-hyperbolic, then the metric space embeds into a tree with ∞ distortion bounded above by 2δ log 2 (2n). The proof proceeds by a linkage argument, i.e. by invoking the definition of hyperbolicity at different scales along chains of points. By virtue of the embedding theorem, one can argue that hyperbolicity is a measure of the "treeness" of a given metric space. It has been shown in [1,2] that various real-world data sets, such as Internet latencies and biological, social, and collaboration networks are inherently treelike, i.e. have low hyperbolicity. Thus, by Gromov's result, these real-world data sets can be embedded into trees with additive distortion controlled by their respective cardinalities. The cardinality bound might of course be undesirable, especially for very large data sets such as the Internet. However, it has been claimed without proof in [1] that Gromov's embedding can yield a 3-approximation to the NTP, independent of [3].
We note that the assumption of a metric input is not apparent in Gromov's embedding theorem. Moreover, the proof of the theorem does not utilize any metric property. This leads one to hope for bounds where the dependence on cardinality is replaced by a dependence on some metric notion. A natural candidate for such a metric notion is the doubling dimension of a space [15], which has already found applications in learning [17] and algorithm design [15]. In this paper, we present novel upper bounds on the additive distortion of a Gromov embedding, depending only on the hyperbolicity and doubling dimension of the metric space.
Our main tool is a stability theorem that we prove using a metric induced by a Voronoi partition. This result is then combined with the results of Gromov's linkage argument. Both the stability theorem and Gromov's theorem rely on the embedding satisfying a particular linkage condition, which can be described as follows: for any embedding f : (X, d X ) → (X, t X ), and any x, x ∈ X, we have
is a chain of points joining x to x and Ψ is some function of d X . A dual notion is to flip the order of the max, min operations. Interestingly, under the correct objective function Ψ, this leads to the well-studied notion of SLHC. By virtue of this duality, the arguments of both the stability theorem and the scaling theorem apply in the SLHC setting. We introduce a new metric space statistic that we call ultrametricity (analogous to hyperbolicity), and are then able to obtain novel lower bounds, depending only on doubling dimension and ultrametricity, for the distortion incurred by a metric space when embedding into an ultrametric tree via SLHC.
We remark that just by virtue of the duality between Gromov's embedding and the SLHC embedding, it is possible to obtain a distortion bound for SLHC depending on cardinality. We were unable to find such a bound in the existing HC literature, so it appears that even the knowledge of this duality, which bridges the domains of HC and MDS methods, is not prevalent in the community.
The paper is organized as follows. The main thrust of our work is explained in §1. In §2 we develop the context of our work by highlighting some of the surrounding literature. We provide all definitions and notation, including the Voronoi partition construction, in §3. In §4 we describe Gromov's embedding and present Gromov's distortion bound in Theorem 3. Our contributions begin with Theorem 4 in §4 and include all the results that follow: namely the stability results in §5, the improved distortion bounds in §6, and the proof of tightness in §7.
The supplementary material contains (1) an appendix with proofs omitted from the body, (2) a practical demonstration in §A where we apply Gromov's embedding to a bitmap image of a tree and show that our upper bounds perform better than the bounds suggested by Gromov's embedding theorem, and (3) Matlab .m files containing demos of Gromov's embedding being applied to various images of trees.

This paper provides new stability results and bounds on the distortion for the tree embedding of a metric space proposed by Gromov. It also exploits a duality between Gromov's embedding and the single linkage hierarchical clustering. As a consequence, stability and bounds are given for SLHC as well. The bounds are illustrated on simulations.
The paper is very interesting and provides good theoretical results. Such an analysis is welcome in the field. However, I think that it could be re-organized to improve its readibility and impact. There are repetitions which may hinder a smooth reading. (see examples below)  In particular, I believe that the proofs could be put in the supplementary materials (although short sketch proofs can stay and give the main idea and steps). The definitions could be restricted only to what is absolutely needed. There are long math developments which may not be integrated enough into the global reasoning: the reader does not necessarily know why he needs to read this.  This deletion/reorganization would give room for: (i) providing intuitive illustrations of metric spaces, trees, voronoi cells, doubling dimension; (ii) giving the intuition behind the theorems 5, 6, 7, 8 (intuition behind inegalities on D in Th.7,8?) and their importance; (iii) developing Section 7 to a page, (iv) providing more structure to the paper (subsections?), (v) write a short conclusion. For now, the intuition is given in "An overview of our approach" in the introduction, but not step by step in the core of the paper.  Examples of repetitions: - Additive distortion is defined twice, in Introduction and Section 2. - The trivial bound is given in Introduction and Section 4. The introduction is maybe too technical? - Gromov's embedding is a 3-approximation to the optimal tree representation in Introduction and Section 7  Examples of unecessary math background: - many details on multiplicative distortion even though not used. Eg: what is the point of the first paragraph of Introduction and last paragraph of Sec. 2? Could they be shortened? - pages 4 and 5 could be embedded in a global reasoning, to keep the reader interested?


Regret of Queueing Bandits
We consider a variant of the multiarmed bandit problem where jobs queue for service, and service rates of different servers may be unknown. We study algorithms that minimize queue-regret: the (expected) difference between the queue-lengths obtained by the algorithm, and those obtained by a "genie"-aided matching algorithm that knows exact service rates. A naive view of this problem would suggest that queue-regret should grow logarithmically: since queue-regret cannot be larger than classical regret, results for the standard MAB problem give algorithms that ensure queue-regret increases no more than logarithmically in time. Our paper shows surprisingly more complex behavior. In particular, the naive intuition is correct as long as the bandit algorithm's queues have relatively long regenerative cycles: in this case queue-regret is similar to cumulative regret, and scales (essentially) logarithmically. However, we show that this "early stage" of the queueing bandit eventually gives way to a "late stage", where the optimal queue-regret scaling is O(1/t). We demonstrate an algorithm that (order-wise) achieves this asymptotic queue-regret, and also exhibits close to optimal switching time from the early stage to the late stage.

The paper considers a variant of the multi-armed bandit problem in which it may take several slots to obtain a reward, and so arriving jobs accummulate in queues. Such problems have been heavily studied in the "queueing theory" field, where long-term performance is analyzed assuming that all the parameters are known. In this paper the authors assume that the "service rate" (i.e. the time it on average takes to obtain the reward) is unknown and needs to be learnt. The paper defines a queue-regret and analyzes its behavior. It establishes an asymptotic lower bound and designs an algorithm that achieves this bound up to a poly log factor. The analysis is non-trivial, as there are two phases: the queues first possibly build up until sufficient learning took place to stabilize them.
While the analytical results are impressive, the paper is not easy to read and confuses the reader. The appendix includes theorems and proofs for a multi-queue setting, while the main body is devoted to a single-queue setting. The introduction and abstract mix both and do not convey properly what the paper is about. Below are my concerns and some suggestions.  1. There is a lot of confusion w.r.t. queueing terminology and setting. I applaud the authors to step outside of their comfort zone, but the paper would hugely benefit from getting more familiar with queueing theory. It is essential to properly explain in the introduction the queueing problem you consider, e.g. do you consider a discrete or continuous time, how many servers you consider, are these pre-emptive, do they serve in parallel, how many queues are there, are the jobs routed immediately after their arrival (to a queue) or when a server becomes idle (to a server), are all the arriving jobs of the same class? In fact, there are a lot of queueing settings, and your single-queue model I believe best fits the problem "routing to parallel servers", which you however not even mention. You consideration of multiple queues is often described as "multi-class" (there are multiple classes of jobs that arrive, each class has a particular set of parameters - in your case \lambda's and \mu's). Other problems, e.g. scheduling, are related but fundamentally different. This will help your paper to have a larger impact as bridging the gap between machine learning and queueing theory.  2. Although catchy, your title is not specific enough. It is not clear what "queueing" means - as there are many queueing settings one could consider. It is not clear what "bandits" means - bandit feedback, bandit algorithm (which?), bandit model (which?).  3. There has been recent research on formulating queueing systems using bandit models. These are of at least 3 kinds: (1) job scheduling problems (see Gittins book 1989, and several papers by Ayesta and his co-authors since 2009), where each server selects which job to serve; (2) congestion control problems (e.g. Jacko & Sanso 2012 and Avrachenkov et al 2013), where users sending TCP packets are trying to learn the optimal transimission rate the Internet allows; (3) routing problems (see several papers by Glazebrook, Nino-Mora and their co-authors) where each job select which server to join. This literature may help you clarify which queueing system you consider in your paper (and perhaps give ideas for future research).   4. There has been recent research on queueing systems with unknown service rate, which you have completely ignored. It is essential that you review those papers and clarify what has already been done and what is different in your paper. E.g., Guo and Matta 2002, Zikos and Karatza 2011, Debo, Veeraraghavan 2014 and the papers citing these.  5. The abstract need to make clear what problem you consider. Is mentioning of the bandit model really essential there? I also don't think the sentence starting "A naive view of this problem would suggest..." is appropriate for an abstract.  6. I would encourage you to describe in the introduction the full problem (with multiple queues). This is because you should also refer to the related literature of the full problem. After you describe the queueing problem in the introduction, make very clear how the bandit model is used to model it: an "arm" could be a single job, or a queue of jobs, or a server (or many other things); how many arms are pulled every time; what happens if there is no job to serve; what what happens if the job is not completed (no reward), what are the possible rewards. In Section 3 you can then limit to case U=1 if you wish to present this problem only.  7. Your Assumption 1 (Initial state) seems very strong to me. In practice it will almost never be satisfied. There needs to be a discussion on whether it is essential for your results and what happens when it is not satisfied.  8. Several references have typos in the titles (e.g. kl-ucb, thompson, c|mu, bayesian)  9. Several unclear phrases are listed below - line 42: "incorporating queueing behavior into the MAB model is an essential challenge" ->  it sounds like it has not been done before, but see the literature listed above - line 48: "reward equals job service" -> "reward is 1 of job service is completed in the current slot, 0 otherwise." - line 50 is correct only in the single server setting - line 57: "formally" - make it more formal (are you interested in finite-time or asymptotic results?) - line 62-64: this is far from obvious, mostly because the model is not clear yet - line 66: "unable to even stabilize" - this will depend on the parameters (for some it may be always stable) - line 77: "below zero" -> why not "negative"? - line 105: "up to a log log t factor" -> maybe more precisely "up to a 1 / log log t factor"  - line 148 (and elsewhere): "arrival rate" and "service rate" are appropriate in a continuous time setting. Here it would be more precise to use "arrival probability" and "service completion probability" - line 154 (and in the appendix): "service offered" - define... what are the possible values? - equation above line 157 is repeated (1) - line 195: Defition 1 is wrongly worded. It is also not clear whether being alpha-consistent is problem-instance dependent - line 205: "bandit algorithms" - which of them do you mean? - line 213 (and lines 261, 284): give a link where the full proof can be found - line 225: it is not clear if you start from t=0 or t=1 - line 238-240: repertitive to 187-189 - line 267: "the the" - line 316: you mention comparison, without linking to where it can be found - "stability region" - define - "crowdsourcing example described in the introduction" - I haven't found one - line 436: the sentence is not finished. - Theorem 16 is different from Theorem 1. It is not obvious how Theorem 1 is obtained, nor it is mentioned that the proof of Theorem 16 is also the proof of Theorem 1. - notation \mu* and \mu_max is used inconsistently - Figure 3 and "Additional discussion" should be moved to the main body if space permits - Please add arrows in Figure 3 to indicate which curve refers to which policy, as it is not clear (I only see 4 curves)


Coupled Generative Adversarial Networks
We propose coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images. In contrast to the existing approaches, which require tuples of corresponding images in different domains in the training set, CoGAN can learn a joint distribution without any tuple of corresponding images. It can learn a joint distribution with just samples drawn from the marginal distributions. This is achieved by enforcing a weight-sharing constraint that limits the network capacity and favors a joint distribution solution over a product of marginal distributions one. We apply CoGAN to several joint distribution learning tasks, including learning a joint distribution of color and depth images, and learning a joint distribution of face images with different attributes. For each task it successfully learns the joint distribution without any tuple of corresponding images. We also demonstrate its applications to domain adaptation and image transformation.

The paper proposes a method for learning generative models of pairs of corresponding images belonging to two different domains (e.g. the RGB image of a scene and the corresponding depth image). The method is based on two adversarial networks with partially shared weights. The generative networks share the weights that map the noise to an intermediate code but have separate weights that map from the intermediate code to each image type. This induces the model to generate pairs of corresponding images even when it is never trained with corresponding image pairs. The authors evaluate the proposed method with several image datasets, and also provide a demonstration of how it can be applied to domain adaptation and cross-domain transformation problems.
I am in two minds about the paper. On the one hand the idea seems very interesting and powerful as it does not seem to rely on pairs of corresponding training images. The authors make a good effort of demonstrating and analyzing the capabilities of the approach in several ways (I especially like the unsupervised domain adaptation results). Visually the results look promising. (Although from the mostly qualitative evaluation it is not clear to what extent the models are overfitting to the training data, and e.g. for the RGB-D experiments I am finding it very hard to say anything more than that the generated pairs look superficially plausible.)  But on the other hand, it seems to me that the task is not well defined. Since the the model is never presented with corresponding image pairs there is actually nothing in the training data that establishes what “corresponding” means. The only pressure for the network to establish a sensible correspondence between images in the two domains comes from the particular weight sharing constraint which allows each network only limited capacity to map from the shared intermediate layer to the two different types of images (the evaluation in the paper uses networks that use only one or two non-shared layers). This may be appropriate, and work well, for domain pairs that differ mostly in terms of low-level features (e.g. faces with blonde / non-blonde hair, or RGB and D images, as in the paper). But it makes me wonder how easy it would be to impose just the appropriate capacity constraint for domain pairs where the correspondence is at a more abstract level and/or more stochastic (e.g. images and text).


Dynamic matrix recovery from incomplete observations under an exact low-rank constraint
Low-rank matrix factorizations arise in a wide variety of applications -including recommendation systems, topic models, and source separation, to name just a few. In these and many other applications, it has been widely noted that by incorporating temporal information and allowing for the possibility of time-varying models, significant improvements are possible in practice. However, despite the reported superior empirical performance of these dynamic models over their static counterparts, there is limited theoretical justification for introducing these more complex models. In this paper we aim to address this gap by studying the problem of recovering a dynamically evolving low-rank matrix from incomplete observations. First, we propose the locally weighted matrix smoothing (LOWEMS) framework as one possible approach to dynamic matrix recovery. We then establish error bounds for LOWEMS in both the matrix sensing and matrix completion observation models. Our results quantify the potential benefits of exploiting dynamic constraints both in terms of recovery accuracy and sample complexity. To illustrate these benefits we provide both synthetic and real-world experimental results.

The authors analyze low rank matrix estimation in a simple dynamic setting where one of the low rank factors varies smoothly across time intervals. This a first attempt at theoretical understanding of dynamic models in high dimensional matrix estimation. The authors derive consistent error bounds for the proposed dynamic model using a weighted squared error minimizer under a non-convex rank constraint.
My primary concerns are listed below: A. Proof 1. a) In line 379 (supplementary material) Theorem 2.3 from 5 cannot be directly used to establish RIP for \sum_t \sqrt{wt} At(\Delta)  as \sum_t wt ||At(\Delta)||_2^2  NOT = ||\sum_t \sqrt{wt} At(\Delta)||_2^2. However, as theorem 3.1 requires bound on only the former, the proof in [5] can be extended. The proof needs some work though. b) Lemma B.2 inequality is reversed.   B. Theory 2. The results are derived for exact global solution to (2) which is a non-convex optimization. The paper is incomplete without the suggested future work on analyzing alternating minimization. I believe the analysis will follow through with a bit of linear algebra machinery, but the exact expression of the joint error term arising from Thm 3.4 and 3.8 and the weighted power iteration is more useful towards understanding the tradeoff of sample complexity and accuracy using the dynamic estimation in practical implementations.  3. For the proposed estimator, simple extensions to the dynamic model analyzed (in line 108) can drastically increase the number of parameters to be estimated using cross validation as explained below: a) Since the e^t are all assumed to be Gaussian with mean zero and a common standard deviation of sigma_2 (independent of t). This setting essentially boils down to the standard matrix estimation problem, where the samples have different noise levels, i.e. each y^t as samples from X^d with Gaussian noise of sigma_1+\|U\|(d-t)sigma_2. b) At a high level, the optimum w^t essentially would weight the fit to the samples based on the noise level at y^t.The algorithm for computing optimum w^t requires estimating a  \kappa parameter (which depends on sigma_1,sigma_2) using cross validation. c) Even for a simple extension of having different sigma_t for e^t, the optimum estimation of w^t would require essentially estimating all the different noise variances using cross validation which can be computationally demanding. d) As I understand, in a naive implementation the optimum w^t require knowledge of noise level of y^t (which is restrictive). A different estimator to avoid this dependence on noise variant might be more useful here. The following work might be relevant in this regard: Klopp, Olga, and Stéphane Gaiffas. "High dimensional matrix estimation with unknown variance of the noise." (2015).  C. Experiments: What does it mean to get results at a particular \kappa? The following comments assume that the authors solve for w^t using the QP in eq 5 using \kappa=\kappa in plots. 4. Figure 1: a) I am not able to judge the significance of the differences from baseline in both experiments. Since the authors have already run the experiments 5 times, I suggest they add error bars to the Figure 1. b) Optimum \kappa for Fig 1a does not match the theoretical values for matrix sensing even for data simulated from modeling assumptions. c) It is very difficult to understand this figure and the corresponding text. The key point I think is that the optimum \kappa for min recovery error increases as sigma 1 increases. This can be highlighted by using labels to indicate the optimum \kappa as (LOWEMS), \kappa\sim1e-2 as baseline 2 and \kappa > 10e2 as baseline 1.  5. Figure 3a: Since both gamma and kappa are chosen from cross validation. The result does not necessarily indicate that the performance improvement is from choosing kappa optimally. I suggest controlling for changes in gamma in this experiment.  (suggestion) 6. Figure 2 highlights the trade-off between sample complexity and accuracy achieved by LOWEMS. However, I think a better plot to illustrate the effect would be to plot two figures (a) for each model m plot the sigma 2 vs the # of samples to for error convergence (say # samples so that the error is below 1.1*min_error(m) ), and (b) for each model m plot the sigma 2 vs the #min_error(m)   MINOR: Line 24: "most popular approach in recent years has focused on the use of nuclear norm minimization as a convex surrogate": This is true for theoretical analysis. The low rank constraint and alternating minimization are more popular in practice. Line 50: use Big O notation or state that C^i are constants. Line 108: equation number Thm 3.4: State that D_1 is also a constant and \delta_1,\delta_2r are RIP constants from Def 3.3 not any positive constants  [EDIT]: I have read the author response and my review remains unchanged. In summary, there are some interesting ideas and potential reference value in being the first analysis of theoretical properties of a dynamic matrix completion model. However, the model analyzed is too simplistic, and the significance of the paper is further undermined by the lack of analysis for a working algorithm (like alternating minimization).  I am currently unable to judge the significance of empirical improvements without the error bars (although the authors response claims to include them).


Stochastic Variational Deep Kernel Learning
Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.

This paper presents a model (SV-DKL) that use deep neural networks as feature extractors and apply additive base kernels to the output features. The deep neural networks are pre-trained followed by a training procedure of the additive base kernel and neural network parameters in cohesion. The parameters are optimized by variational inference and the learning has intriguing runtime properties. The model shows promising results compared to SVM and deep neural networks with and without GP.
This is a well-written paper with only a small number of grammatical and presentation errors, e.g. there is no numbering (a, b, c, d) in figure 2.   The theoretical basis for this work is sound and explained well. However, the experimental section lacks some detail on the training of DNNs and the pre-training. From the level of detail given it will be hard to reproduce the results.   Furthermore the results are not too convincing. The authors claim that "flexible deep kernel GPs achieve superior performance" to DNNs and SVMs. The difference in the classification results doesn't seem as convincing. It would be nice to have a more detailed discussion on whether the comparison is fair. Currently the DNNs in the stand-alone model and the DNN+GP/SV-DKL are all parameterized equally, meaning that the GP and SV-DKL additions both have more parameters. Furthermore the DNN+GP addition is trained independently, thus it will not have the same flexibility as the training of SV-DKL. Would the DNN produce better results if it were just defined with more parameters? As stated earlier it is not clear how the DNN is trained, but it would be beneficial to the training of the DNN with regularization (e.g. dropout); hence the KL term in the ELBO will work as a regularizer for the SV-DKL.  In conclusion, this is a well-written paper, but it needs to argue a better case in order to be a convincing novel contribution.


Optimistic Gittins Indices
Starting with the Thomspon sampling algorithm, recent years have seen a resurgence of interest in Bayesian algorithms for the Multi-armed Bandit (MAB) problem. These algorithms seek to exploit prior information on arm biases and while several have been shown to be regret optimal, their design has not emerged from a principled approach. In contrast, if one cared about Bayesian regret discounted over an infinite horizon at a fixed, pre-specified rate, the celebrated Gittins index theorem offers an optimal algorithm. Unfortunately, the Gittins analysis does not appear to carry over to minimizing Bayesian regret over all sufficiently large horizons and computing a Gittins index is onerous relative to essentially any incumbent index scheme for the Bayesian MAB problem. The present paper proposes a sequence of 'optimistic' approximations to the Gittins index. We show that the use of these approximations in concert with the use of an increasing discount factor appears to offer a compelling alternative to state-of-the-art index schemes proposed for the Bayesian MAB problem in recent years by offering substantially improved performance with little to no additional computational overhead. In addition, we prove that the simplest of these approximations yields frequentist regret that matches the Lai-Robbins lower bound, including achieving matching constants.

The authors propose a new anytime bandit algorithm based on an efficient approximation of the discounted Gittins index. There are two main ideas. The first is to increase the discount rate as a function of time, which saves the strategy from linear regret. The second is to approximate the Gittins index via a truncation after which the signal noise drops to zero.   The primary technical contribution is a theoretical analysis showing the frequentist asymptotic optimality of the strategy when the lookahead is just 1.
*************************************************** POST REBUTTAL.  Thanks for the correction. I agree this makes things OK again. Nice paper. ***************************************************   Overall the paper is reasonably well written. I get the feeling the authors removed a lot of intuition to make the paper fit. I'm sympathetic to this plight, but still wish there was a little more there. Even simple things like noting that the effective horizon for discount gamma = 1 - 1/t is O(t) would be helpful. Also, in Examples 3.1/3.2 it would be nice to see some approximate solutions that reveal the strategy has a form approximately the same as some existing strategies (see minor comments).  Technically the paper is generally OK, but I do have some possibly serious issues described below. The authors should focus on these in their response to this review  to increase my score.   * It is claimed at the start of Section 4 that because the strategy is asymptotically optimal in the Lai & Robbins sense that it is also asymptotically Bayesian optimal. Please justify this claim. As far as I know the UCB strategy is not asymptotically Bayesian optimal, but it is optimal in the Lai & Robbins sense. The first index-strategy that is Bayesian optimal is by Lai in '87 (you cite it). I do believe your strategy gets both, but I don't understand the argument.  * In L397 in the supplementary material. I don't understand the > in the display. What if q = 1/10 and p = 1/2. Then the first logarithmic term is negative. I tried a little to fix this result and could not. This lemma seems quite critical and I wonder if this is where the improvement is coming from that allows some log factors to be shaved in the confidence interval essentially (I am talking about the discussion in L223).  Besides the theoretical results there are also some experiments. I found these to be a bit of a tease because (a) the performance is excellent and (b) we only get to see examples from a two (random) experiments. One Gaussian and one Bernoulli. It's nice to see the quantiles of the distribution, but I would have preferred to see the expected regret over a larger range of problems. Eg., the worst-case regime. Instead the focus is on a very easy problem with few arms and a distribution for which most of the arms are well separated. That said, the results are impressive.   Overall I find the paper very interesting. The technical issues force me to lower my score and I wish the execution were better (crisp intuition/experiments). See below for some minor comments.    Minor comments:  * It might be worth discussing the approximations given in Examples 3.1 and 3.2 in a little more depth. I did not investigate the former, but the latter turns out to be quite interesting. Solving the inequality and doing some naive approximation suggests that the optimal lambda has approximately the form  lambda = sqrt(2sigma^2 log(t sigma^2)   (here I have chosen gamma = 1 - 1/t as recommended). If the prior is the flat improper Gaussian prior, then in round t we have  sigma^2 = 1/N_i(t-1) and one recovers a well-known index, which at least for the Gaussian case has now extremely tight, but not quite optimal (in finite time),  regret guarantees. There are also asymptotic guarantees, see [1]. Note that this is basically the algorithm that Lai showed was Bayesian optimal (replace t by n).  * L39. I think Berry & Fristedt showed already that the Bayesian strategy is index-based only if geometric discounting and infinite horizon are assumed.  * L76. This reference is wrong, although that paper does study the case where the horizon is not known, so may also be relevant. I guess you mean [2]  * L216. Not everything that decays faster than 1/t is summable (eg., 1/(t log(t))  * The regret analysis included in the main body is all totally standard, while the interesting stuff is deferred to the appendix. An odd choice?  * L208: s_t, s_t' are introduced. Are they ever used?  * A bit petty, but I find some of the notation a bit displeasing. For example, T is used for the non-random horizon while k_i(t) is used for the  random number of times arm i has been chosen.  * Is it written which priors were used by Thompson sampling/Bayes UCB in the experiments?  * For what prior is Theorem 1 proven? (looks like flat beta prior, but is it written?). Can you get asymptotic optimality for any prior? Also see [1] for a more general results on BayesUCB   References:  [1] On Bayesian index policies for sequential resource allocation. Kaufmann, 2015 [2] Regret analysis of the finite-horizon Gittins index strategy for multi-armed bandits. Lattimore, 2016


Double Thompson Sampling for Dueling Bandits
In this paper, we propose a Double Thompson Sampling (D-TS) algorithm for dueling bandit problems. As its name suggests, D-TS selects both the first and the second candidates according to Thompson Sampling. Specifically, D-TS maintains a posterior distribution for the preference matrix, and chooses the pair of arms for comparison according to two sets of samples independently drawn from the posterior distribution. This simple algorithm applies to general Copeland dueling bandits, including Condorcet dueling bandits as a special case. For general Copeland dueling bandits, we show that D-TS achieves O(K 2 log T ) regret. Moreover, using a back substitution argument, we refine the regret to O(K log T + K 2 log log T ) in Condorcet dueling bandits and most practical Copeland dueling bandits. In addition, we propose an enhancement of D-TS, referred to as D-TS + , to reduce the regret in practice by carefully breaking ties. Experiments based on both synthetic and real-world data demonstrate that D-TS and D-TS + significantly improve the overall performance, in terms of regret and robustness.

The paper develops Thompson-sampling style algorithms for the interesting and relevant problem of dueling bandits, where the best arm has to  be learnt using only pairwise preference feedback. The key idea in the main algorithm is to sample twice, and independently, the unknown model from the posterior distribution over models, and essentially pit the best arms for both models against each other. This has the advantage of promoting exploration in early phases and converging to playing the optimal arm against itself in later phases. The authors derive detailed regret bounds for the algorithm for both the Condorcet winner and the more general Copeland winner ground truth models, and perform numerical comparisons of the proposed algorithm with existing, Upper-Confidence-Bound (UCB) style strategies.
EDIT: I have read the author response.  The paper takes on the challenging task of adapting Thompson sampling-style Bayesian-inspired approaches to the dueling bandit problem, and thus I feel that its contributions are significant. The dueling bandit problem, being a true partial monitoring problem (not all actions give nonzero statistical information), is in a sense harder than standard bandits, and hence requires more care in designing the exploration-exploration tradeoff. Although this is not the first time Thompson sampling has been proposed for the problem (e.g., RCS is an earlier attempt), the smart idea of sampling twice and following it up by a UCB-elimination step is quite novel, and supplemented by fairly competitive analytical guarantees. I believe this would open up paths for more in-depth looks at Thompson sampling approaches to solve classes of dueling bandit problems.   Detailed comments and observations are as follows.   *    Without the Condorcet winner assumption, the algorithm is shown to give a O(K^2logT) regret bound, where as the CCB algorithm of Zoghi et al. gives O(K^2 + KlogT) regret, and SCB gives an even better regret bound of O(KlogKlogT) with certain restrictive assumptions.  Yet, in the experiments, these algorithms seem to be performing quite poorly w.r.t. the proposed algorithm which is counter intuitive, even for 500-armed case D-TS seems to be performing better than CCB/SCB (Fig. 4(d)). Could the authors suggest an explanation for this observation?  * Comparing with existing algorithms with different notions of regret for the dueling bandit problem: Are the regret definitions used in this paper directly comparable to that of some of the existing dueling bandit algorithms (e.g. RUCB, Merge-RUCB, RMED, specially those which are based on Condorcet winner assumptions)? It seems that the notion of regret used in this paper is not directly reducible to that used in these other papers with a Condorcet winner assumption.  * One wonders how the regret performance gets affected if one just uses double Thompson sampling for pulling the two arms (a^(1) and a^(2) in Algorithm 1) from the whole set of K arms, instead of selecting them from the restricted set as described in line 11 and 14 of Algorithm 1. In other words, how much significant is the restrictive sampling (that is used for pulling the pair of arms) in terms of bounding the regret?   * Experiments:                  -It might be worth comparing how the algorithm's performance scales w.r.t. other algorithms, e.g. Merge-RUCB, CCB/SCB etc. with increasing K.   -It might also be worth comparing different algorithms with certain other class of pairwise preference matrices, e.g. pairwise preferences with a Borda winner, with total ordering, or preferences generated from random utility model etc.   - It is not clear why ECW-RMED has higher standard deviations of regret in comparison to the proposed method since in the later case both the arms are sampled from some posterior distribution.


Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates
Non-negative matrix factorization is a popular tool for decomposing data into feature and weight matrices under non-negativity constraints. It enjoys practical success but is poorly understood theoretically. This paper proposes an algorithm that alternates between decoding the weights and updating the features, and shows that assuming a generative model of the data, it provably recovers the groundtruth under fairly mild conditions. In particular, its only essential requirement on features is linear independence. Furthermore, the algorithm uses ReLU to exploit the non-negativity for decoding the weights, and thus can tolerate adversarial noise that can potentially be as large as the signal, and can tolerate unbiased noise much larger than the signal. The analysis relies on a carefully designed coupling between two potential functions, which we believe is of independent interest.

The paper shows recovery guarantees for non-negative matrix factorization under adversarial noise and unbiased noise. The algorithm proceeds by alternatively decoding and encoding, which correspond to updating weight matrix and updating feature matrix respectively. The analysis requires some conditions including linear independence among feature columns, independent sampling of weights for different features, balanced second moments of the weights and some good initialization.
The recovery guarantee for NMF with mild conditions is very important for the machine learning community. This paper provides a solution under three main conditions as follows.  1) The conditions, A0 and A1, on the feature matrix are very mild. 2) The condition of the weights, A2, is also reasonable except for the independent requirement.  3) The condition for a good initialization, A3. However, it is unclear if this initialization can be achieved.  The proof technique seems to be novel, i.e., different from the popular techniques used for alternating minimization of matrix completion or the tensor methods. Therefore, it will probably be useful for the community. Therefore, I vote poster-level for novelty and oral-level for potential impact.   I have two concerns for this paper,  1. It seems that the theorems hold only with some (high) probability, which is not stated in the theorem. The reason is that the encoding step uses empirical expectation rather than the population expectation. Even if provided large enough samples as claimed on Line 643 and Line 647 in the appendix, as long as the number of samples is finite, the theorems will not hold almost surely without more assumptions on the weights.   2. If I understand Sec. 5 correctly (Sec. 5 is a little harder to read), the goal of this section is to provide solutions when the weights don't satisfy Assumption A2. However, according to Theorem 6, A^* is required to satisfy Assumption A0, which implies A^*D will not satisfy A0. Therefore, we can't apply Algorithm 1 afterwards to get A^* as claimed on Line 293. This will invalidate the whole Sec. 5.   Due to the above two concerns, I just vote poster-level for the technical quality.   The paper is not hard to understand, so I vote poster-level for clarity.   Some minor comments: 1. At the end of the paper Line 292-297, the authors mention how to initialize. However, it lacks reference and clear statement. 2. It will be better to provide some experimental results to verify the theoretical analysis, even if it is on synthetic data.  3. The lemma number is missing near Line 411 in the appendix.  4. Line 294, pontentially -> potentially  ========= To authors' feedback.  I have read the feedback. The feedback clarified my concerns.