Barzilai-Borwein Step Size for Stochastic Gradient Descent
One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization methods, the common practice in SGD is either to use a diminishing step size, or to tune a step size by hand, which can be time consuming in practice. In this paper, we propose to use the Barzilai-Borwein (BB) method to automatically compute step sizes for SGD and its variant: stochastic variance reduced gradient (SVRG) method, which leads to two algorithms: SGD-BB and SVRG-BB. We prove that SVRG-BB converges linearly for strongly convex objective functions. As a by-product, we prove the linear convergence result of SVRG with Option I proposed in [10], whose convergence result has been missing in the literature. Numerical experiments on standard data sets show that the performance of SGD-BB and SVRG-BB is comparable to and sometimes even better than SGD and SVRG with best-tuned step sizes, and is superior to some advanced SGD variants.

The authors give a new analysis of SVRG. It allows using "Option I" (taking the final iterate of the inner iteration), as is done in practice. They also propose to use a scaled version of Barzilai-Borwein to set the step-sizse for SVRG (and heuristically argue that this could also be useful for classic stochastic gradient methods too). Their experiments show that this adaptive step-size is competitive with fixed step-sizes.
Note that I increased my score in light of the experiments discussed in the author response.   I previously reviewed this paper for ICML. Below I've included some quotes from my ICML review that are still relevant. But first, I'll comment on some of the changes and lack of changes after the previous round of reviewing:  1. The authors have removed most of the misleading statements and included extra references and discussion, which I think makes the paper much better.  2. One reviewer brought up how the quadratic dependence on some of the problem constants is inferior to existing results. I'm ok with this as having an automatic step-size is a big advantage, but the paper should point out explicitly that the bound is looser. (This reviewer also pointed out that achieving a "bad" rate under Option I is easy to establish, although in this case I agree with the authors that this contribution is novel).  3. The paper is *still* missing an empirical comparison with the existing heuristics that people use to tune the step-size. The SAG line-search is now discussed in the introduction (and note that this often works better than the "best tuned step size" since it can increase the step-size as it approaches the solution) but there is no experimental comparison and the MISO line-search heuristic is still not even discussed. To me this is a strange omission: if the proposed method actually works better than these methods then including these comparisons only makes the paper stronger. Even if the proposed method works similarly to existing methods, it still makes the paper stronger because it shows that we now have a theoretically-justified way to get the same performance. Not including these comparisons is not only incomplete scholarship, but it makes the reader think there is something to hide. (I'm not saying there is something to hide, I'm just saying there are only good reasons to include these experiments and only bad reasons not to!)  4. One reviewer pointed out a severe restriction on the condition number in the previous submission, which has been fixed.  --- Comments from old review ---  Summary:  The authors give a new analysis of SVRG. It allows using "Option I" (taking the final iterate of the inner iteration), as is done in practice. They also propose to use a scaled version of Barzilai-Borwein to set the step-sizse for SVRG (and heuristically argue that this could also be useful for classic stochastic gradient methods too). Their experiments show that this adaptive step-size is competitive with fixed step-sizes.  Clarity:  The paper is very clearly-written and easy to understand (though many grammar issues remain).  Significance:  Although several heuristic adaptive step-size strategies exist in the literature, this is the first theoretically-justified method. It sill depends on constants that we don't know in general, but I believe is a step towards black-box SG methods.  Details:  Independent of the SVRG/SG results, the authors give a nice way to bound the step-size for the BB method.  Normally, BB leads to a much faster rate than using a constant step-size, but in the SVRG setting your theory/experiments are just showing that it does as well as the best step-size (which is good, but it isn't better than the best step size).  Finally, the paper would be much stronger if it compared to the two existing strategies that are used in practice: 1. The line-search of Le Roux et al.  where they increase/decrease an estimate of L. 2. The line-search of Mairal where he empirically tries to find the best step-size.  However, I don't think that the proposed approach would actually work better than both of these methods (but these older approaches don't have any theory).


Hypothesis Testing in Unsupervised Domain Adaptation with Applications in Alzheimer's Disease
Consider samples from two different data sources {x i s } ∼ P source and {x i t } ∼ P target . We only observe their transformed versions h(x i s ) and g(x i t ), for some known function class h(·) and g(·). Our goal is to perform a statistical test checking if P source = P target while removing the distortions induced by the transformations. This problem is closely related to domain adaptation, and in our case, is motivated by the need to combine clinical and imaging based biomarkers from multiple sites and/or batches -a fairly common impediment in conducting analyses with much larger sample sizes. We address this problem using ideas from hypothesis testing on the transformed measurements, wherein the distortions need to be estimated in tandem with the testing. We derive a simple algorithm and study its convergence and consistency properties in detail, and provide lower-bound strategies based on recent work in continuous optimization. On a dataset of individuals at risk for Alzheimer's disease, our framework is competitive with alternative procedures that are twice as expensive and in some cases operationally infeasible to implement.

This paper introduces a new method for domain adaptation that relies the Maximum Mean Discrepancy (MMD). Transformations of both the source and target distributions are learned while jointly minimizing the MMD between the transformed versions. This yields a statistical test of whether the predefined function class is rich enough to have transformations that make the two distributions indistinguishable, and this test is shown to be consistent.  The method is illustrated on batches of data from an Alzheimer's disease study, showing that shifting the distribution achieves some improvements in analytical performance.
The paper presents an interesting and smart way of performing covariate shift by aiming to make the two distributions indistinguishable by minimizing MMD. The paper however could benefit of more clarity and completeness so it can make impact.   In terms of the applicability of this approach, the authors talk about the importance of being able to perform statistical tests and not just optimize the performance of a classifier. The bounds that they derive for their statistical test is useful to know how big the sample size should be to perform an appropriate shift. However, this doesn't say much on how it affects the scientific questions asked in the experiment, which need different statistical test. From the two sites where the data originates, we might (for example) have two types of patients. How does this shift help us with identifying them? Table 1 shows the answer to a similar question where the hippocampal volume is better predicted by the morphed CSF data, but more tests such as this one are required to believe that the method is useful. What if the question of interest was to find the CSF proteins that are most predictive, how would the shifting of the covariates affect our ability to answer the question?  More experiments, as well as a better explanation of the problem would make the paper stronger. For example, what if the data from the two sites were unevenly distributed such as they had different proportions of the possible phenotypes. The variance we would see would not be a consequence of the measurement only, but also reflects population variance. The proposed measure could however homogenize the two batches and introduce some error. Perhaps these questions are hard to answer theoretically but more experimental results could help.  It would be interesting to also mention how this method could be applied to more than two batches.


Nested Mini-Batch K-Means
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested mini-batches, whereby data in a mini-batch at iteration t is automatically reused at iteration t + 1. Using nested mini-batches presents two difficulties. The first is that unbalanced use of data can bias estimates, which we resolve by ensuring that each data sample contributes exactly once to centroids. The second is in choosing mini-batch sizes, which we address by balancing premature fine-tuning of centroids with redundancy induced slow-down. Experiments show that the resulting nmbatch algorithm is very effective, often arriving within 1% of the empirical minimum 100× earlier than the standard mini-batch algorithm.

The paper presents an accelerated implementation of mini-batch k-means algorithm, with supported good empirical performance.
Technical quality:  It seems the nested-batch method is likely to introduce overhead by keeping all previously sampled points in memory, especially since mini-batch k-means is usually run for many iterations? And the computational cost of checking whether a point is already sampled grows as the number of iteration grows as well. How did this not seem to have an effect in your experiments, as comparing to the original mini-batch algorithm?  The experiments in Fig. 2 may be a little misguiding: it shows that nested-mini-batch achieves same level of k-means cost faster than the other compared methods; however, this may only mean that it plateaued faster. As time increases, it's possible that the other algorithms will achieve a lower k-means cost eventually (they reach a plateau with a lower k-means cost). The authors do not discuss whether the experiments reveal this feature of nested-mini-batch k-means.   Novelty:  The paper mostly adapts the idea of distance bounding using triangle inequality (Elkan) to the setting of mini-batch k-means.  Potential impact: This paper may have marginal contribution to the existing mini-batch method.  Clarity and presentation: The paper is well written and the ideas are clearly presented.


Dialog-based Language Learning
A long-term goal of machine learning research is to build an intelligent dialog agent. Most research in natural language understanding has focused on learning from fixed training sets of labeled data, with supervision either at the word level (tagging, parsing tasks) or sentence level (question answering, machine translation). This kind of supervision is not realistic of how humans learn, where language is both learned by, and used for, communication. In this work, we study dialog-based language learning, where supervision is given naturally and implicitly in the response of the dialog partner during the conversation. We study this setup in two domains: the bAbI dataset of [23] and large-scale question answering from [3]. We evaluate a set of baseline learning strategies on these tasks, and show that a novel model incorporating predictive lookahead is a promising approach for learning from a teacher's response. In particular, a surprising result is that it can learn to answer questions correctly without any reward-based supervision at all.

This paper proposes to learn question answering models using feedback in the form of short textual exchanges. This can be seen as learning from a very primitive form of dialog between a teacher (who provides the questions and some feedback) and another student (who provides answers that are correct with some fixed probability).   The paper proposes alterations to two existing datasets: the artificial bAbI dataset, and a movie QA dataset, in order to incorporate 10 different types of supervision from the teacher and student. In some settings, an explicit reward is given when the student answers correctly. An augmentation to the memory network architecture is proposed that achieves strong performance on the bAbI dataset, and moderate performance on the real-world MovieQA dataset.
The proposed research direction of exploring models that learn from different types of conversational feedback is interesting and, to my knowledge, novel. The presentation of the task is well-motivated, and could lead to some interesting future research directions. The paper is fairly well-written.   When proposing a new task, it is often difficult to determine how challenging to make the task. By far the most interesting task proposed is task 7, where no reward-based feedback is given. However, it would be very nice to have more variations of this task. For example, one could have a varying number of script templates for correct and incorrect examples, vary the length of the responses, or have more realistic ‘noisier’ responses (ie. where the feedback is hidden in a larger response). This would isolate the language component of the feedback for learning QA policies – it is important to determine what kind of language feedback can be learned from, and this paper only explores one such aspect (whether additional information is included from the teacher in the response), and this only in the setting with reward-based feedback.  In my view such tasks would be more useful than some of the other proposed tasks (for example tasks 9 and 10, detailed below).  I am slightly concerned that the remainder of the tasks proposed, those with reward-based feedback, are too easy, and that the variations between tasks are too small. However, in some ways this is good for a new task, as one can isolate the cases where a proposed model under-performs (which is hard to do with complex tasks). It is impossible to know with certainty whether the tasks are too easy until more diverse models (other than just memory networks) are tried.  A crucial point of the paper is that the learner (ie. the memory network model being learned) is not the one actually interacting with the teacher. Instead, the model simply observes the conversation between the teacher and some other student, and learns to answer the questions from varying forms of feedback. It should be made more explicit in the abstract and introduction that the model being learned in this paper is not a dialog agent, but is a question answering system. This becomes evident much later in the paper, however for clarity this should be addressed earlier.  The formulation of tasks 9 and 10, where the student asks for help, is also unclear. In the description, it is stated that ‘the learner must ask for help in the dialog’. This seems to imply that the learner must *learn* to ask for the help of the teacher. However, the learned model is not interacting with the teacher at all – it is simply observing the other student answering this question. Thus, this is indeed very similar to tasks 3 and 5 (which is reinforced by the similarity of the results in Tables 1 and 2). This can be inferred from later sections in the paper, but is not clear enough at the outset.  The proposed memory network extension incorporating forward prediction is interesting, and its success is well-justified in section 5. The experiments are well-conducted – in particular, the results on the MovieQA dataset are crucial to demonstrate that this supervision approach can apply to real-world tasks, and provides room for improvement for future models. The fact that all of the models and baselines are memory network-variants is a slight negative. Also, it would be good to have some kind of error estimates – the models that should achieve random performance (FP model on task 1, RBI model on task 7) exhibit significant variance on the bAbI dataset (fluctuating from 20-29%), which implies that there could be some variance in the remainder of the results. This appears to be less of a problem on the MovieQA dataset.  Overall, this is a fairly good paper that makes an interesting contribution in terms of new tasks and models. There is some concern that these tasks are not difficult enough, as well as some issues with clarity in the paper (detailed above), which should be addressed by the authors. However, I could see the proposed tasks becoming a bAbI-style toy task used for model justification, so that these models can later be used for more difficult versions of the tasks.  Small comments: Line 57: put a period after [14] in related work


Learning Bound for Parameter Transfer Learning
We consider a transfer-learning problem by using the parameter transfer approach, where a suitable parameter of feature mapping is learned through one task and applied to another objective task. Then, we introduce the notion of the local stability and parameter transfer learnability of parametric feature mapping, and thereby derive a learning bound for parameter transfer algorithms. As an application of parameter transfer learning, we discuss the performance of sparse coding in selftaught learning. Although self-taught learning algorithms with plentiful unlabeled data often show excellent empirical performance, their theoretical analysis has not been studied. In this paper, we also provide the first theoretical learning bound for self-taught learning.

This paper shows an excess risk bound in a transfer learning setting, with the main emphasis to be the scenario where the data have a latent sparse structure (as in sparse coding). The basic bound, Theorem 1, is proved using an error decomposition into three parts, each of which can be bounded using by-now standard techniques. The main work is establishing (under various assumptions) that a sparse-coding based method learns (from unlabeled ``source'' data) a dictionary that is sufficiently close to the optimal dictionary, and moreover, that under sufficiency proximity, that the sparse codes are stable under dictionary perturbations. The latter result, Theorem 2, was already proved by Mehta and Gray, but the authors offer a variation on this earlier result. They further show (and this constitutes the main technical work of the paper) that under certain assumptions, a notion they call the $k$-margin is large enough to apply the sparse coding stability result.
\documentclass{article} \usepackage[in]{fullpage} \usepackage{amsmath} \usepackage{amssymb} \usepackage{amsfonts}  \begin{document}   This paper has an interesting direction and synthesizes a number of results from different previous analyses of sparse coding methods to develop a excess risk bounds for a two-layer (sparse coding and then linear prediction) predictor. While I think this very much is a worthwhile pursuit, I have some concerns with the final (rather implicit) version of Theorem 1 after plugging in the various results from Section 3 (which is concerned with sparse coding, stability, and dictionary learning). I also have some technical concerns that I'll describe shortly. If there are good responses to these concerns, I will increase the score for technical quality (I thought the techniques used in this paper and those it uses from others are highly non-trivial, but the technical score is low right now due to potential correctness issues).  Overall, I found Theorem 1 a straightforward result (assuming a sparse coding stability result and the right error decomposition, the proof is not difficult). I did find the analysis in Section 3 to be techniquely involved and to carefully draw from a number of previous results in this space in order to control the dictionary estimation error and the margin. I did not check the proof of Theorem 3 carefully. However, it is concerning that the failur probability in Theorem 3 does not appear to decay with either $N$ or $n$, but only with the dimension $d$. Practically speaking, this may not be an issue, but theoretically, it limits the power of the results as the confidence is fixed with respect to $N$ and $n$. A similar issue occurs elsewhere, but potentially affecting the rate, as I now describe.  The most pressing issue is how to interpret Theorem 1 under the setting considered in Section 3 (Assumptions 1 through 4). For Theorem 1 to give a non-trivial bound, we must have $\|\hat{\theta}_N - \theta^*\|_{1,2} = o(n^{-1/2})$ due to the last term of (8). Therefore, looking at (43), in the case that $\tau = \frac{1}{4}$, we need $d = \Omega(n^{2/5})$ (and for $\tau = \frac{1}{2}$ we need $d = \Omega(n^{1/5})$). But $d$ is the ambient dimension, not a free parameter! Therefore, we really do not have the ability to play with $d$ in the bound (this did not seem like a nonparametric setting to me). Therefore, let us turn our attention to the earlier appearing equation (41).  If (41) holds as stated, then what prevents us from taking $M, M' \rightarrow \infty$ and thereby concluding that as long as $N$ is polynomial in $d$, then, for all $\varepsilon > 0$, with probability 1 we have $\|\hat{D}_N - D^*\|_{1,2} < \varepsilon$. This feels quite suspect, and so I looked at Theorems 3 and 4 of Arora et al.; their statements say the rate is $d^{-\omega(1)}$,, which I take to mean we have $d^{-h(N)}$ where $\lim_{N \rightarrow \infty} h(N) \rightarrow \infty$. Therefore, to make Theorem 1 useful (say, with a rate of $o(1)$), you need to ensure that $d^{-h(N)} = o(n^{-1/2})$; here I assume that you have made some choice of $N$ to depend on $n$, and typically $N = \Omega(n)$ since unlabeled source data is ``cheap''. Now, for $d^{-h(N)} = o(n^{-1/2})$ to hold seems easy enough if $h(N)$ is linear in $N$, but if it is logarithmic in $N$, one has to be more careful. Therefore, you need to dig into Arora et al.~and present their $\omega(1)$ bound with some additional precision. Please include a response to this in the rebuttal.   Regarding Theorem 2, given how different the RHS of (25) is as compared to the corresponding term in Theorem 4 of Mehta and Gray, you should have a proof which at least outlines what changes when doing the analysis under a bound on a $\|\cdot\|_{1,2}$ perturbation instead of their $\|\cdot\|_{2,2} = \|\cdot\|_2$ perturbation. Also, the scaling as $\lambda^3$ is quite disappointing, especially considering that the scaling is only as $\lambda$ in Theorem 4 of Mehta and Gray. Do you think this can be improved, or are perturbations in the $\|\cdot\|_{1,2}$ norm really that much more perilous?    It would be helpful to provide a short proof of (17), because in your proof of Theorem 1, it seems that wherever you wrote $\|\hat{\theta}_N - \theta^*\|$, you actually meant to write $\|\hat{\theta}_N - \theta^*\|_{1,2}$, since the latter is what actually appears in the theorem statement. But in (17), it is not clear to me if you should be able to get dependence merely on $\|\hat{\theta}_N - \theta^*\|_{1,2}$ rather than the (typically larger) $\|\hat{\theta}_N - \theta^*\|_{2,2}$. Please clarify this in the rebuttal.   Finally, please have your paper checked for English grammatical issues.     \subsection*{Minor points}  \begin{enumerate} \item    I think (using Hoeffding's inequality, a martingale version not being needed here) the last term in (13) should be $L_\ell R_\mathcal{W} R_\psi \sqrt{\frac{2 \log(2/\delta)}{n}}$, since each term is in $[0, 2 R_\psi]$. Also, I don't see how you get $\log(4/\delta)$ since your confidence setting is $\delta/2$. It isn't clear to me that these discrepancies all result from using Azuma-Hoeffding rather than (non-martingale) Hoeffding. \item    Assumption 4 is unclear. Do you mean to say that $\xi$ is zero-mean sub-Gaussian with support contained in the ball in $\mathbb{R}^d$ of radius $\sigma / \sqrt{d}$? That is, the case of bounded noise (which is hence sub-Gaussian)? \item    On line 256, do you mean to set $M = 5 \tau$ (rather than $M = 5 c$)?  \end{enumerate}      \subsection*{Update after authors' rebuttal}  Regarding my issue with Theorem 3, which is that the failure probability $\delta_{t,,\lambda}$ does not decrease with $n$ or $N$: The authors' response makes sense to me and I agree that this failure probability should be independent of $n$ and $N$. Also, I think in the high-dimensional setting this is acceptable due to the failure probability's exponential decay with $d$, practically speaking.  With regards to my issue with the application of Theorem 1 using the sparse coding application of Section 3:  Because the authors mentioned that they were not quite correct in some of the additional statements they made beyond (41), but that (41) itself is correct, I had another look at Arora et al., and was frustrated with the imprecision of that paper in terms of using the phrase "high probability" without explaining technically what they actually mean; my conclusion is that I do not know whether or not the authors are correctly interpreting and applying the results of Arora et al., but I am willing to give the authors the benefit of the doubt here. HOWEVER: I want the authors to be very precise on the exact result they are using from Arora et al. (more precise, it seems, then the exposition, of Arora et al. themselves).  Moreover, I had thought there may have been an issue with the authors own tuning of their parameters in Section 3, because the application of Theorem 1 in the case of sparse coding appears to give a trivial bound for different settings of $\tau$ unless we have the following limitations (I consider only the cases of $\tau = 1/4$ and $\tau = 1/2$): \begin{itemize} \item If $\tau = 1/4$, then we only apply the bound in the regime $n = O(d^{5/2})$. This seems acceptable, since $n$ is in the target domain, and we often do not assume all that much data in the target domain. \item If $\tau = 1/2$, then we only apply the bound in the regime $n = O(d^5)$. This seems even less of a restriction. \end{itemize} Still, I would like the authors to update their paper to be up-front about these (currently implicit) restrictions.  So, my technical concerns are adequately addressed, owing to some optimism on the part of the authors' use of the results of Arora et al.    \end{document}


Deep Learning Games
We investigate a reduction of supervised learning to game playing that reveals new connections and learning methods. For convex one-layer problems, we demonstrate an equivalence between global minimizers of the training problem and Nash equilibria in a simple game. We then show how the game can be extended to general acyclic neural networks with differentiable convex gates, establishing a bijection between the Nash equilibria and critical (or KKT) points of the deep learning problem. Based on these connections we investigate alternative learning methods, and find that regret matching can achieve competitive training performance while producing sparser models than current deep learning strategies.

The paper reformulates learning in NNs as a game, and demonstrates a correspondence between Nash equilibria and KKT points.  I enjoyed reading the paper and am extremely sympathetic to its aims.  However, I have concerns about: (i) whether the zanni's play a significant role; (ii) whether the experiments evaluate anything meaningful, and (iii) whether the results apply to the neural nets that people actually use.   If these are suitably addressed, I will modify my scores accordingly. Detailed comments follow.
Detailed comments  1. Line 36 introduces h(x) = \phi(\theta x) where \phi is a transfer function, which I guess is intended to model the nonlinear activations in the output layer of the (single-layer) NN. However, neither h nor \phi show up in the remainder of the section. Incorporating \phi breaks Theorems 1 and 2 since the loss \ell(\phi(\theta x), y) will not be a convex function of \theta for most choices of function \phi. I guess the authors meant to say “One linear layer learning games”?  2. Generality of results Regarding Theorems 4 and 5, the ancillary lemmas in the appendix assume that the activation functions f_v are convex and differentiable. This means that the most commonly used functions in practice (ReLU, maxout, max-pooling, sigmoid, and tanh units) are all ruled out along with many others.  The results in the paper therefore apply in much less generality than a cursory reading would suggest. The authors should clarify which NNs the results apply to, preferably in the introduction.   More broadly, can the results be extended to the neural nets used by practitioners? If not, then the title "Deep learning [x]" is a little misleading.  3. Constraints Minor point: The L1 constraint is disconnected from the rest of the paper. Meaning, the reasons for *specifically* choosing L1 are unclear to me. One could just as easily have chosen, say, L2 and used different OCO algorithms than EWA. Why highlight this particular constraint? Sparsity is mentioned a few times, but sparsity is conceptually orthogonal to the paper’s main contribution.  4. Zannis Line 225 states “the usual process of backpropagating the sampled (sub)gradients [computes] the best response actions for the zannis and the antagonist and the resulting affine utility for the protagonists.” This raises two major questions.  First question: Is it necessary to introduce zanni’s into the game in the first place? Since their best responses are given by backpropagation, one could directly construct the loss/utility for each protagonist by suitably backpropagating the loss specified by the antagonist — this recovers the path-sum gradient game introduced by Balduzzi in [2].  Continuing, line 210 states “we investigated […] independent protagonist agents at each vertex against a best response antagonist and best response zannis”. So the experiments don’t use the additional degrees of freedom introduced into the setup by introducing zannis as agents in their own right. Instead, the authors presumably just plug in backprop for the zannis. So, again, do the zannis add anything real to the setup?  I’m actually curious how the games run when all players including zannis are “alive” and update their responses using one of Algorithms 2-4. For example, does the game still converge to the same or comparable local optima to when the zanni best responses are computed via backprop? How much does convergence slow down?  Second question: What do the experiments evaluate? Line 229 states “to investigate the plausibility of applying independent expert algorithms at each vertex [of] deep learning models”. But applying SGD, RMSProp, Adam etc to backpropagated errors is *exactly* how NNs are trained. Since the zanni’s best responses are backpropagated errors, it seems the only difference between standard NN training and the experiments in the paper is that the authors impose constraints on the weights. However, the effect of the constraints (for example, changing the set's size) is not investigated. And of course the effect of constraints isn’t really what the paper is about. So what do the experiments tell the reader?  Please clarify this point. Otherwise, aside from the RM result, the experiments come down to running standard optimizers on an NN on MNIST and observing that they work.  As an aside, it is very interesting that Regret Matching (RM) is competitive with RMSProp et al in the experiments and yields sparser solutions. However, this needs to be investigated more thoroughly before it can be considered a robust result. It is also largely orthogonal to the main concerns of the paper.  Minor point: Is zanni standard terminology? I’ve never encountered it before and find it quite unpleasant. Please consider alternatives.


The Robustness of Estimator Composition
We formalize notions of robustness for composite estimators via the notion of a breakdown point. A composite estimator successively applies two (or more) estimators: on data decomposed into disjoint parts, it applies the first estimator on each part, then the second estimator on the outputs of the first estimator. And so on, if the composition is of more than two estimators. Informally, the breakdown point is the minimum fraction of data points which if significantly modified will also significantly modify the output of the estimator, so it is typically desirable to have a large breakdown point. Our main result shows that, under mild conditions on the individual estimators, the breakdown point of the composite estimator is the product of the breakdown points of the individual estimators. We also demonstrate several scenarios, ranging from regression to statistical testing, where this analysis is easy to apply, useful in understanding worst case robustness, and sheds powerful insights onto the associated data analysis.

The paper considers the robustness of composite estimators, estimators that apply one estimator to sections of the data set and another estimator to the output of the first estimator. The problem is important due to the fact that big data sets are hard to curate and composite estimators are often applied to them to simplify data analysis. The notion of robustness is quantified by the breakdown point, which is the minimum fraction of data points that need to be significantly modified in order to significantly modify the output. The breakdown point of a composite estimator is usually the product of the breakdown points of its component estimators. The authors apply this result to analyze the robustness of existing composite estimators (such as median of medians) and propose ways to increase their robustness.
Technical Quality: The technical quality is good. The proofs appear to be sound and the remarks touch on important issues. The main text does not give experimental results. My main concern is how to couple this work with qualities of estimators that are traditionally desirable in statistics. For example, people often want unbiasedness or statistical efficiency. The applications section talks about how to make estimators more robust, but can we get estimators that are robust and have a low MSE?  Novelty: The paper is fairly novel. It builds upon previous basic work on the robustness of estimators via the breakdown point by defining a new version of the breakdown point and extending to composite estimators.   Potential impact: I think that this work is important for real-world data analysis. In some fields existing algorithms have many steps giving complicated composite estimators, but the problem of error propagation and robustness may not be given due attention. This work may also be useful for companies that don't spend the time to curate their business data. My only concern is the difficulty of computing the breakdown point for more complicated estimators. The paper only considers percentiles and other median-based estimators.   Clarity and presentation: The paper is well-organized and well-written, and the authors communicate the ideas clearly. The proofs were easy to follow. However, there are a few typos and grammar errors. Also, to better show the applicability of the work, I think that the simulations ought to be added to the paper and the complete proofs pushed to the supplement if needed.


Convex Two-Layer Modeling with Latent Structure
Unsupervised learning of structured predictors has been a long standing pursuit in machine learning. Recently a conditional random field auto-encoder has been proposed in a two-layer setting, allowing latent structured representation to be automatically inferred. Aside from being nonconvex, it also requires the demanding inference of normalization. In this paper, we develop a convex relaxation of two-layer conditional model which captures latent structure and estimates model parameters, jointly and optimally. We further expand its applicability by resorting to a weaker form of inference-maximum a-posteriori. The flexibility of the model is demonstrated on two structures based on total unimodularity-graph matching and linear chain. Experimental results confirm the promise of the method.

A convex training formulation for a two-layer model p(z|y)p(y|x), i.e., the model Z <- Y <- X with X denoting the input, Y the latent space, and Z the output, is proposed and investigated. Contrasting existing work, the authors only require tractable MAP inference on p(y|x), which is available for graph-matching type models or generally any integer linear program with a totally unimodular constraint set.  A classical approach for two-layer models is the bi-level optimization based solution of p(z|\argmax_y p(y|x)). I.e., we first optimize the encoding model over y in an `inner' optimization, and infer the solution over the decoder in a second `outer' step. Training the parameters of those models is challenging due to the fact that small perturbations of the encoder model lead to the same prediction of the latent representation y, which is assumed to be discrete. Hence gradients for training the models are zero almost everywhere.  The presented approach plugs the first-order optimality conditions of the `inner' optimization into the `outer' optimization, by using Lagrange multipliers and a saddle-point formulation. The resulting problem is non-convex in the training parameters, and a semi-definite programming (SDP) formulation is proposed, which yields a convex relaxation after dropping the rank constraint.  Hence the authors state that learning depends on only the MAP operation over the latent space which is assumed to be efficient.  The proposed approach is demonstrated on the task of transliteration and image inpainting. The method outperforms the shown baselines.
Review summary:  A nice technique to circumvent the issues arising from bi-level optimization is suggested and applied to probabilistic modeling. However quite a few details are missing to assess the quality of the proposed solution, e.g., additional assumptions beyond the given ones might be required (see point 1 below), important aspects are not explained (see point 3 below), and the experimental evaluation is a little weak. See below for details.  - Technical quality: derivations are sound but some additional assumptions should be required to ensure efficiency - Novelty: novel approach to two-layer modeling - Potential impact: hard to assess since details regarding efficiency of the proposed paper are not stated - Clarity: lacking explanation of some important aspects   I'll adjust my score based on the author feedback.  Review comments:  1. Let's assume z to be discrete. The decoder probability defined in Eq. (4) depends on the function G. To ensure proper normalization of the decoder distribution, G should correspond to the log-partition function. In the proposed formulation (e.g., Eq. (15)), maximization over its dual, i.e., the entropy restricted to the probability simplex, is required. Isn't tractability of this maximization task required/assumed then? In particular, the resulting program seems to be a quadratic term plus entropy terms and simplex constraints. Can the authors comment on how they solve this task efficiently? Sec. 5 does not seem to contain any details.  2. The following is not part of the proposed approach but the authors might nonetheless ensure correctness. In l.113, it is stated that the partition function `\Omega(Ux)' is replaced by an upper bound `\max -y'Ux'. The authors may want to check whether the maximization is indeed an `upper bound'. If I'm not wrong  \Omega(Ux) = \log\sum_y\exp(-y'Ux) = -y^\ast'UX + \log\sum_y\exp(-y'Ux + y^\ast'Ux) (*)  where y^\ast = \arg\max_y -y'Ux.  However the log term in (*) can either be positive or negative, hence the maximization is not really a bound. Did I overlook something?  3. Why are the model parameters bounded? There were no constraints in the original formulation illustrated in Eq. (7). Hence it's counterintuitive to see constraints in Eq. (10) and the following ones. I think any reader would appreciate a more sophisticated argument beyond the stated one: `to simplify the presentation'. Could the authors provide a few more details regarding the reason for introducing those bound constraints? Is feasibility an issue?  4. Can the authors comment on the quality of the proposed baselines? The method is compared to an approach from 2010 on the transliteration task. It might be useful to provide more recent baselines as well.  5. The authors may also want to comment on the computational complexity of the proposed method. Usage of an SDP formulation seems computationally expensive and not necessarily scalable to larger problem sizes.  6. Moreoever I'm wondering whether the authors compared the proposed SDP approach to gradient descent on the objective given in Eq. (15).  Minor comments:  - The authors could mention that the ^\prime operator denotes the transpose.


Noise-Tolerant Life-Long Matrix Completion via Adaptive Sampling
We study the problem of recovering an incomplete m × n matrix of rank r with columns arriving online over time. This is known as the problem of life-long matrix completion, and is widely applied to recommendation system, computer vision, system identification, etc. The challenge is to design provable algorithms tolerant to a large amount of noises, with small sample complexity. In this work, we give algorithms achieving strong guarantee under two realistic noise models. In bounded deterministic noise, an adversary can add any bounded yet unstructured noise to each column. For this problem, we present an algorithm that returns a matrix of a small error, with sample complexity almost as small as the best prior results in the noiseless case. For sparse random noise, where the corrupted columns are sparse and drawn randomly, we give an algorithm that exactly recovers an µ 0 -incoherent matrix by probability at least 1 − δ with sample complexity as small as O (µ 0 rn log(r/δ)). This result advances the state-of-the-art work and matches the lower bound in a worst case. We also study the scenario where the hidden matrix lies on a mixture of subspaces and show that the sample complexity can be even smaller. Our proposed algorithms perform well experimentally in both synthetic and real-world datasets.

This paper studies recovering an incomplete low rank matrix with noise, in an online fashion. The model is such that the learner are allowed to adaptively choose to unifomrly sample d features, or do a full measurement. Two noise models are studied, namely the bounded deterministic noise model, and the sparse random noise model. The authors provide recovery guarantees for both cases (robust recovery for the first case and exact recovery for the second case). The result matches or improved the state-of-art. However, notice that results in literature are for passive learning, where the authors assumes active learning/adaptive sampling, so the comparison is just illustrative.
This paper studies recovering an incomplete low rank matrix with noise, in an online fashion. The model is such that the learner are allowed to adaptively choose to unifomrly sample d features, or do a full measurement. Two noise models are studied, namely the bounded deterministic noise model, and the sparse random noise model. The authors provide recovery guarantees for both cases (robust recovery for the first case and exact recovery for the second case). The result matches or improved the state-of-art. However, notice that results in literature are for passive learning, where the authors assumes active learning/adaptive sampling, so the comparison is just illustrative.  This paper is well written and organized nicely. The contribution is in generally interesting. My main conerns are  (1) whether the adaptive sampling model used is significant/practical. The authors motivate this model via Netflix rating prediction where the company can sequentially send out questionnaries to each person for movies. While this is a valid rational if the adaptive sampling involves a moderately higher sampling rate, it does not justify the proposed algorithm where one may request a column being full observed (equivalent to ask a user of Netflix to watch all movies and rate!!) (2) The sparse random noise model, where the noise is assumed to be sparse and iid following a distribution. In literature, most sparse noise are modeled as arbitrary and adversarial, and hence the model studied by the authors are much easier. Even in such case, the tolerable noise is very low (indeed the cardinality of the support of the noise must be lower than the cardinality of the observed entries). It is not clear to me how much this can be improved. The lower bound presented does not cover this issue either.  Some minor comments: 1. subsubsection 3.1.2 should be a subsection, and 3.1.3, 3.1.4 combined should be a separate subsection too. 2. To avoid being misleading, the authors may want to mention that the lower bound does not rule out the possibility of improving the tolerance of the noise. 3. In theorem 6, the result depends on h very mildly, is there any intuition behind it?


The Power of Adaptivity in Identifying Statistical Alternatives
This paper studies the trade-off between two different kinds of pure exploration: breadth versus depth. We focus on the most biased coin problem, asking how many total coin flips are required to identify a "heavy" coin from an infinite bag containing both "heavy" coins with mean ✓ 1 2 (0, 1), and "light" coins with mean ✓ 0 2 (0, ✓ 1 ), where heavy coins are drawn from the bag with proportion ↵ 2 (0, 1/2). When ↵, ✓ 0 , ✓ 1 are unknown, the key difficulty of this problem lies in distinguishing whether the two kinds of coins have very similar means, or whether heavy coins are just extremely rare. While existing solutions to this problem require some prior knowledge of the parameters ✓ 0 , ✓ 1 , ↵, we propose an adaptive algorithm that requires no such knowledge yet still obtains near-optimal sample complexity guarantees. In contrast, we provide a lower bound showing that non-adaptive strategies require at least quadratically more samples. In characterizing this gap between adaptive and nonadaptive strategies, we make connections to anomaly detection and prove lower bounds on the sample complexity of differentiating between a single parametric distribution and a mixture of two such distributions.

This paper studies a heavy-coin detection problem where there are (potentially infinite) many coins, of which $1-alpha$ percent are light coins with parameter $theta_0$ and $alpha$ percent are heavy coins with parameter $theta_1(>theta_0)$. This paper proposes an adaptive strategy that is based on estimation (of theta's)-and-exploration (to find heavy coins), but also improves upon it using a doubling trick. Hardness results for other alternatives (particularly the baseline estimation-and-exploration without the doubling trick) are used as evidence towards the novelty of this contribution.
This problem is interesting and there is a history of discussion on similar topics in EE/ML/TCS proceedings particularly in the multi-armed bandits community. There are certain novelty in this paper that can be a nice contribution. However, my biggest problems with the paper are in the organization of the materials and a lack of discussions with some key papers that are very relevant.  The authors discussion on the background is not entirely comprehensive. For example, eq (1) is fairly common knowledge in the multi-armed bandits dating back to Lai and Robbins 1985. A very relevant literature for discovery of all heavy coins in a finite-arm case is studied in Bubeck et al., ICML 2013. Additionally, these papers do not require knowledge of $theta_0, theta_1$ either and further allow each coin to have its own parameter while the result is adaptive with the actual realization (e.g., if the parameters fit the model in the paper under review, then an equivalent of (1) can be obtained). Nonetheless, the paper under review bears some novelty because the other two papers focused on discovery of all "heavier coins" whereas this paper is able to find one heavy coin using only amortized complexity.  The connection to hypothesis test (P1) is also unclear -- unless the authors suggest that there are many distributions that are either H_0 or H_1 and one wants to find one distribution that belongs to H_1. I am not sure if this is the idea of a hypothesis test. Also, why would I only want just one example distribution from H_1 instead of finding all distributions belonging to H_1?  The second equality in the line above (2) contains a typo. It makes sense to write dP/dQ as Radon-Nikodym derivative when P is absolutely continuous on Q; however, it makes no sense to write dP in isolation in the integrand. I would instead write it in terms of pdf or pmf.  Section 3 is tedious and confusing. I would put them in the appendix, but only state why each term should appear in the result (instead of explaining how one should go about proving the theorems). The paragraph below 3.2 is particularly confusing. I directly skipped to Table 1.   In Section 4, what is SPRT? Algorithm 3 may need better introduction as my understanding of it is an informed guess based on the term doubling trick. I did not check for the proofs of Theorem 5; it seems to be a core part of the paper but it is also at the very end of the paper. I can read it further upon request.  As for novelty/impact if I only perceive this paper as an algorithm that finds one heavy coin with sample complexity that equals the amortized cost to find all heavy coins (and assuming the proofs are correct): it has some novelty. However, the setting is rather restrictive (there are only two types of arms). As the authors suggest, neither this paper nor the Carpentier and Valko paper (assuming arm parameter having exponential distribution) covers the full flexibility of the original best-arm identification problem. These assumptions on the arm distribution may hinder real-world applications. The author should also revise the paper for clarify and emphasize on the algorithms and some real-world motivations.


Contextual semibandits via supervised learning oracles
We study an online decision making problem where on each round a learner chooses a list of items based on some side information, receives a scalar feedback value for each individual item, and a reward that is linearly related to this feedback. These problems, known as contextual semibandits, arise in crowdsourcing, recommendation, and many other domains. This paper reduces contextual semibandits to supervised learning, allowing us to leverage powerful supervised learning methods in this partial-feedback setting. Our first reduction applies when the mapping from feedback to reward is known and leads to a computationally efficient algorithm with near-optimal regret. We show that this algorithm outperforms state-of-the-art approaches on real-world learning-to-rank datasets, demonstrating the advantage of oracle-based algorithms. Our second reduction applies to the previously unstudied setting when the linear mapping from feedback to reward is unknown. Our regret guarantees are superior to prior techniques that ignore the feedback.

The authors consider a "semi-bandit" learning setting, in which a learner selects a group of L actions from a base set of K actions across a sequence of rounds. On each round, nature determines rewards for each base action, and reveals these values for the L base actions chosen by the learner (the semi-bandit feedback).   The reward gained by the learner for the composite action is some weighted linear combination of the L base rewards, where this weighting might be unknown to the learner (and depends on the ordering of the base actions). The key contribution over previous work is to extend this type of problem to a contextual setting, where the goal is to compete with the best policy in some class.  The authors assume access to a supervised learning oracle, which is able to return the best policy for some finite dataset of contexts, base-rewards, and weightings, and give an algorithm which achieves sqrt{T} regret while making a polynomial number of calls to this oracle.
The closest work to this setting can be found in Kale et al., which considers the same problem and gives similar rates. However, the algorithm is a modification of EXP4 and requires that the learning maintain a distribution over all policies in the policy class. In contrast, the authors use a technique very similar to that of Agarwal et. al. (https://arxiv.org/pdf/1402.0555v2.pdf) to solve a corresponding optimization problem via coordinate descent (and calls to the supervised learning oracle) while maintaining a distribution over policies with sparse support. The authors claim that the Kale et al. work also has the limitation that the weight vector must be w = 1. However, this is a somewhat dubious claim in the known-weights setting (unless I am mistaken, it seems that one could simply construct -w_l y_t(a_{t,l}) as the base-action loss for the Kale et al. algorithm).    Overall, the big ideas of this paper are adaptations of the ideas in Agarwal et. al. However, the analysis is still quite involved and delicate, and ultimately the problem considered is an important one due to applications in online content recommendation.


Unsupervised Learning of Spoken Language with Visual Context
Humans learn to speak before they can read or write, so why can't computers do the same? In this paper, we present a deep neural network model capable of rudimentary spoken language acquisition using untranscribed audio training data, whose only supervision comes in the form of contextually relevant visual images. We describe the collection of our data comprised of over 120,000 spoken audio captions for the Places image dataset and evaluate our model on an image search and annotation task. We also provide some visualizations which suggest that our model is learning to recognize meaningful words within the caption spectrograms.

The paper presents a multi-modal neural network that learns similarities between images (or VGG objects therein) and audio captions that Turkers produced for these images. The idea is that a network would learn what "car" looks like in the audio, if the image shows a "car". While the problem of semantic "grounding" has been around for a long time, the intersection of "language and vision" and deep learning presents a new angle to this problem, which the present paper (and other works) are trying to exploit.
This is interesting work that is pointing into the right direction, but a few aspects of this paper are a bit problematic:  1) It would have been useful (or interesting) to use a corpus that has existing text captions, and either have users re-speak the text captions, or collect additional captions. The data collections seems generally well thought-out, but why was the Places205 data set used? Because of the focus on "objects"? Prompted speech (such as collected here) is not "spontaneous", otherwise the WSJ recognizer would not have given 20 % WER (this aspect is irrelevant for the purpose of this paper, though, I think). Typically, multiple captions are being generated for a single image. Has this been done here as well? Or is there only a single caption for each image? Why?   2) The presentation of the speech/ audio work could be a bit clearer: put all WER results (including DEL/INS/SUB) in a table, present an analysis of the most frequent words for each error type. How much data (in time and/ or words) is 100 captions? < 15 minutes? Is it possible to diagnose more clearly what the difference between the Google and Kaldi recognizers are? How big is the silence portion of the data? How was the vocabulary size (27,891) determined without manual transcription? Was there some post-segmentation of any kind to remove excess silence? (If there is any?) Which fraction of the data was clipped to 10 seconds? Which fraction was padded? Would it be possible to "sub-sample" frames, by iteratively dropping frames that are very similar to their neighbor, until 1000 frames have been reached? Is it necessary to use a CNN, rather than a simple DNN? Does it help? Has VTLN/ feature adaptation been applied? You could get those from the Kaldi recipe, I presume. (Note that the last few items are suggestions and certainly not required in the first publication in the topic)  3) The model seems a bit simplistic, although average pooling and a DNN/CNN architecture is certainly a viable first approach. I like the analysis of drawing parallels in the evaluation to see what the level of performance is with respect to other corpora and/ or tasks, but there are just too many differences. It might have been better to try more variants of the present model to gain insight into what factors influence the performance, and which ones don't (e.g. CNN vs DNN, silence segmentation approach, features, etc.). While the overall story and presentation of the work is good, the third sentence in Figure 2 contains BASEBALL twice. The first time, it is relevant (according to the similarity profile), the second time, it is not. This effect should be analyzed. Is this a feature of the model? Or an artifact that would disappear with better training etc?   It is good that the authors plan to release their data (maybe even with a baseline system).  The references seem good (the authors should check for other relevant work before publication, though).  Finally, the paper has many small typos and grammar errors - please fix.


Consistent Estimation of Functions of Data Missing Non-Monotonically and Not at Random
Missing records are a perennial problem in analysis of complex data of all types, when the target of inference is some function of the full data law. In simple cases, where data is missing at random or completely at random [15], well-known adjustments exist that result in consistent estimators of target quantities. Assumptions underlying these estimators are generally not realistic in practical missing data problems. Unfortunately, consistent estimators in more complex cases where data is missing not at random, and where no ordering on variables induces monotonicity of missingness status are not known in general, with some notable exceptions [13,18,16]. In this paper, we propose a general class of consistent estimators for cases where data is missing not at random, and missingness status is non-monotonic. Our estimators, which are generalized inverse probability weighting estimators, make no assumptions on the underlying full data law, but instead place independence restrictions, and certain other fairly mild assumptions, on the distribution of missingness status conditional on the data. The assumptions we place on the distribution of missingness status conditional on the data can be viewed as a version of a conditional Markov random field (MRF) corresponding to a chain graph. Assumptions embedded in our model permit identification from the observed data law, and admit a natural fitting procedure based on the pseudo likelihood approach of [2]. We illustrate our approach with a simple simulation study, and an analysis of risk of premature birth in women in Botswana exposed to highly active anti-retroviral therapy.

This paper is concerned with the problem of deriving consistent estimators for datasets which have fields that are missing non-monotonically and not at random (MNAR). The authors frame this problem using the language of graphical models by introducing a chain graph depiction of the problem, propose a solution using inverse probability weighting and suggest an estimation technique which uses pseudo-likelihood estimation.  A small set of synthetic experiments on a simple synthetic dataset are carried out to evaluate the proposed technique empirically.
I found the paper to be well written and fairly easy to follow overall. However, I have two concerns regarding this work:  1. The lack of discussion around related work makes it difficult to fully judge the contributions of the paper to the causal inference community. For example, the non-monotone MNAR  setting has been studied, and a solution has been proposed using inverse propensity weighting, by Robins et al. (1999) and Vansteelandt et al. (2007). A nice summarization of these approaches is given by Li et al. (2011). A simple discussion of the estimation technique proposed in section 5 to the previously mentioned work would be very helpful. The proposed work seems unique in that the authors explicitly frame the problem within the context of graphical models, and use propose a pseudo-likelihood approach for estimation. However, it is difficult to judge the novelty of the proposed contributions without additional context.  2. The authors use a very simple synthetic domain for the experimental evaluation. I would have liked to see a larger array of functional forms and dependence structures. It is also not clear from the results that the proposed method achieves a substantial improvement over taking no action at all. In many cases, the adjusted values provide a worse estimate than performing no adjustment at all. It may be the case that more substantial improvement is seen in larger sample sizes but this should be shown within the main text of the paper.  There is a small typo in footnote one where the independence symbol is erroneously defined as the conditional independence. The clarification is also not likely to be necessary given the intended audience.   Overall, this paper presents an interesting idea for an important problem but the lack of framing and a thorough evaluation make it difficult for me to suggest acceptance.   * Robins JM, Rotnitzky A, and Scharfstein D. "Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models." Statistical models in epidemiology: the environment and clinical trials. 1999 * Vansteelandt S, Rotnitzky A and Robins J. "Estimation of regression models for the mean of repeated outcomes under nonignorable nonmonotone nonresponse." Biometrika. 2007 * Li, Lingling, et al. "On weighting approaches for missing data." Statistical methods in medical research. 2011


On Mixtures of Markov Chains
We study the problem of reconstructing a mixture of Markov chains from the trajectories generated by random walks through the state space. Under mild nondegeneracy conditions, we show that we can uniquely reconstruct the underlying chains by only considering trajectories of length three, which represent triples of states. Our algorithm is spectral in nature, and is easy to implement.

The paper proposes a novel approach to estimate the parameters of a mixture of Markov chains. It is based on samples of trajectories with length 3 (the 3-trails) of the finite mixture and uses SVD decompositions and simple linear algebra techniques to recover the parameters from the empirical distribution of these 3-trails. Some synthetic experiments and a real data application are provided.
The paper is globally sound and makes a new contribution to an important topic. However some technicalities need to be addressed and a revised version should be encouraged.   Major remarks: - There is a confusion on whether the Markov chains under consideration are supposed to be stationary or not.  Indeed, the concept of t-trail either requires that the Markov chains under study are stationary or one should specify that all the trails start with same initial distribution, i.e. these trails are observations of (X_1,X_2,X_3) and not (X_s, X_{s+1},X_{s+2}) for some s. I first understood that you adopt the second approach (as you count the parameters of initial distributions as free parameters) but in the real data experiments, you take many (3001)-trails and break them into 3000 overlapping 3-trails (by the way, you need a 3002-trail to obtain these). This causes two different problems: the first is related to stationarity and the second will be developed below. So, by breaking a t-trail into several 3-trails you implicitly assume that the chains follow a stationary regime. This should be stated more clearly. And this should be taken into account when discussing the number of parameters (the stationary distribution for the \ell-th Markov chain will be entirely determined by its transition M^\ell - as soon as some ergodicity conditions are satisfied on M^\ell). To summarize, breaking a trail into several t-trails only makes sense in a stationary regime.  - The second problem induced by breaking a t-trail into several 3-trails lies in the overlap. One could safely consider non-overlapping 3-trails (resulting in a smaller amount of data). Here, the authors choose to rely on overlapping t-trails (at least this is done in the real data experiments). Then, the law of large numbers is satisfied for each Markov chain provided some ergodicity conditions are  assumed on their transition. In particular, the statement that ‘It is easy to see that using O(1/\epsilon^2 log n) sample trails, every entry in O_j for every j is approximated to within an additive +/- \epsilon’ (lines 97-99) is not valid in general when using non independent samples.   - In the synthetic experiments, computation of the error requires to explore all the permutations for matching \mathcal{M} with \mathcal{N}. This is not a trivial task. How do you proceed? (same question applies in the real data experiments, especially when you have L=10 groups). - In the real data experiments with L=10 groups, you assume that the group structure is known and given by the 10 different users. How did you compute the prediction error for L less than 10? What is the truth here? In the case of 10 groups, why not simply computing an adjusted rand index (ARI) to evaluate the classifications (yours and EM) with respect to ‘truth’?  - Synthetic experiments rely on only 11 different runs which is really small. You should go to at least 100 replicates of your experiments for each sample size.  - Synthetic experiments: the authors draw initial distributions s^\ell from the Dirichlet distribution on the simplex (indeed, uniform distribution on the simplex is the Dirichlet distribution with parameters (1,…,1)) and then divide by L to obtain a distribution on the pair (group, initial state). This boils down to assuming that all the groups have equal probability, which could be noticed. More importantly, what happens when the groups are not equally distributed?  - Those results are reminiscent of results obtained by Allman et al. ‘Identifiability of parameters in latent structure models with many observed variables’, Annals of Statistics, 2009. In this paper, the authors rely on a result by Kruskal on 3-ways arrays to prove identifiability of different classes of model. Dimension 3 plays an important role and they also obtain ‘generic identifiability’ results meaning that the set of non-identifiable parameters has Lebesgue measure 0. I believe this reference could be included.      Minor remarks: - Line 82, M^1, …, M^L are more precisely transition matrices of Markov chains (and not Markov chains by themselves).  - Line 110, ‘To appreciate such a need, consider the obviously necessary, though not sufficient, condition that the matrices in M be distinct.’  I don’t see how this is related to the previous sentence.  - Number of parameters (lines 113-114): it seems to me that the set \mathcal{S} corresponds to distributions on the pair (group, initial state), in which case, there are additional constraints on those parameters and they should reduce to L(n-1) free parameters (each one of the L Markov chains has an initial distribution determined by n-1 parameters). In any case, I mentioned above that the general setting of the paper should be the stationary regime in which case initial distribution is not a free parameter anymore.  - Line 228: ‘we consider the first 3001 state transitions that change their state’: what do you mean exactly by ‘that change their state’? Do you mean that you discard data where X_s=X_{s+1}? If yes, why do you do that and why would this be valid?     - line 236:’The prediction error ranges from 1/L to 1’/ I don’t understand that statement. I would say that it ranges from 0 to 1.   - Baseline EM: how do you initialize the algorithm? How many starting points do you use?    Typos:  - Line 83, initial probability is denoted s^\ell(i) while later it becomes s^\ell_i.  - Equation (2): I believe the last factor 2 should be removed.  - Line 176, ‘(P,Q) is a shuffle pair’ should be (\mathcal{P},\mathcal{Q}). - Line 157: matrix R^’ should be R (both instances).  - Line 223-224: E_l is not defined. It is related to ‘error’ above.


Learning Deep Parsimonious Representations
In this paper we aim at facilitating generalization for deep networks while supporting interpretability of the learned representations. Towards this goal, we propose a clustering based regularization that encourages parsimonious representations. Our k-means style objective is easy to optimize and flexible, supporting various forms of clustering, such as sample clustering, spatial clustering, as well as co-clustering. We demonstrate the effectiveness of our approach on the tasks of unsupervised learning, classification, fine grained categorization, and zero-shot learning.

The paper suggests an approach to regularizing neural networks by encouraging their activations to form tight clusters in one of several proposed spaces. A k-means style objective is added to the overall network loss, penalizing deviation of activations in each layer from their closest (layer specific) cluster center. The cluster centers are learned jointly with the network weights.          Three approaches are suggested for clustering: sample clustering, where the vectors to be clustered (at each layer) are the vectorized activations corresponding to different samples in a mini-batch; spatial clustering, where the vectors to be clustered are values across channels corresponding to different locations (and samples) within a convolutional layer's activations (same as sample clustering for fully connected layers); channel co-clustering, where vectors to be clustered correspond to vectorizations of different channels (and samples) within a convolutional layer's activations.          The approaches are demonstrated to have a positive effect on generalization error on multiple datasets and across several tasks, including unsupervised learning on MNIST, classification on CIFAR10 and CIFAR100, and fine-grained classification and zero-shot learning on CUB-200-2011 (birds). Additionally, the clusters obtained provide insight into the structures learned by the network.
         This is a very solid paper all around. The idea to cluster activations and force the network to "stay close" to a restricted set of representations is intuitive, acting as a parsimony constraint that additionally enables interpretability. While solid theoretical motivation and analysis isn't provided, the proposed algorithm feels natural and the experiments are fairly comprehensive and compelling. A broad range of tasks are considered, including unsupervised, fine-grained, and zero-shot learning in addition to standard classification, and the visualizations of cluster structure demonstrate that the cluster centers are meaningful---they convince me that the approach is performing as one would hope.          Some specific points: It would be nice to see a table or chart demonstrating the effect of different choices of cluster size on generalization error (it is mentioned that cross-validation was used, but it would still be good to get a feel for the sensitivity with respect to choice of cluster size). A concern about the method is of course the added computation in each iteration required to update and utilize the clusters. It would be nice to get a sense of how much the proposed approach slows down training (e.g., report time per iteration with and without clustering). Finally, the description of how zero-shot clustering is performed is somewhat vague, and requires consulting the listed references. It would be nice to have a clearer description in the text.


Generating Images with Perceptual Similarity Metrics based on Deep Networks
We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), allowing to generate sharp high resolution images from compressed abstract representations. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric reflects perceptual similarity of images much better and, thus, leads to better results. We demonstrate two examples of use cases of the proposed loss: (1) networks that invert the AlexNet convolutional network; (2) a modified version of a variational autoencoder that generates realistic high-resolution random images.

The manuscript 'Generating Images with Perceptual Similarity Metrics based on Deep Networks' proposes a distance measure between images that is meant to capture perceptual distances well. The distance measure consists of three terms:           1. A standard l2 distance in pixel space           2. A l2 distance in the feature space given by a late layer in a CNN (AlexNet here)           3. An adversarial loss that quantifies the probability with which a discriminator network thinks both images come from the same distribution.                      The manuscript presents two applications for that distance measure:           1. It shows that reconstructions of images from their compressed deep network representations look perceptually much closer to the original image when using the proposed distance measure as a loss function to train a generator network to invert the deep network representations.           2. It shows that images sampled from a VAE are of better perceptual quality when using the distance measure as the reconstruction loss term during training of the VAE.                      Furthermore it demonstrates how one can use the first application to smoothly interpolate between pairs of images by generating images from their linearly interpolated feature representations.
          I think the most important contribution of the manuscript is to describe a method that substantially improves image reconstruction from compressed deep network representations (e.g. 4k dimensions vs. 227^2 * 3 = 150k dimensions), hence showing exciting new ways of images compression using deep networks. In that regard I would have liked an analysis of the compression rate for the reconstructions from the different feature spaces. In particular because the difference in quality between layer conv5 and fc6 doesn't seem too large, whereas there is a 10-fold reduction in dimensionality ((13 x 13 x 256 = 43k vs. to 4k) in the feature representation.                      One factor that should definitely be discussed in the paper is that it appears as if the adversarial prior enforces to use image data from the training set in the reconstruction. This is not necessarily a problem in terms of image compression, but is an important factor: a careful choice of training data might be important depending on what type of images one wants to compress. And the ability to reconstruct arbitrary images might be limited by the limit on available training data.                      The manuscript argues that the described method has potential applications in terms of reconstructing images from measured brain activity. This is indeed an exciting possible application. However, as there probably won't be enough neural data in the foreseeable future to train a generator from, this argument could have been made a lot stronger by showing that the learned generators are useful to reconstruct images from deep representations of new networks. E.g. one could train a linear predictor from a deep VGG representation to a deep AlexNet representation and show that one can still use the generator trained with AlexNet to reconstruct the image from the VGG representation.                      Minor comments/questions:           - Was the comparator trained, it is mentioned that it could be trained in general but it doesn't say if it was in the described experiments. I assume it wasn't.           - To what extend is random comparator different to just using the img + adv loss? There is only one example to compare and there it could be quite similar, meaning that the random representations don't contribute much.           - The manuscript could mention that when training the generator for reconstruction that there is another prior enforced by the architecture of the generator in addition to the learned adversarial prior. This is important because the (convolutional) network architecture substantially limits the images that can be generated.                      In conclusion, I think that in general the manuscript is solid in its methods and arguments and demonstrates findings of importance to the field that will certainly influence future work.


Deep Exploration via Bootstrapped DQN
E cient exploration remains a major challenge for reinforcement learning (RL). Common dithering strategies for exploration, such as '-greedy, do not carry out temporally-extended (or deep) exploration; this can lead to exponentially larger data requirements. However, most algorithms for statistically e cient RL are not computationally tractable in complex environments. Randomized value functions o er a promising approach to e cient exploration with generalization, but existing algorithms are not compatible with nonlinearly parameterized value functions. As a first step towards addressing such contexts we develop bootstrapped DQN. We demonstrate that bootstrapped DQN can combine deep exploration with deep neural networks for exponentially faster learning than any dithering strategy. In the Arcade Learning Environment bootstrapped DQN substantially improves learning speed and cumulative performance across most games.

First of all, let me say I am not an expert in DQN or neural networks, so this review may be a little superficial.  The authors propose a modification of the DQN algorithm for reinforcement learning with neural network function approximation. The objective is to tackle the exploration problem, which is a big issue because the "default" epsilon-greedy approach has provable limitations in all but the simplest of setups. The description of the new approach is sometimes a little vague (see below), but the general approach is to construct several (K - a free parameter) neural networks (or one big neural network with K heads) to approximate the value function and play  each episode by first randomly choosing a network and following its recommendations until the end of the episode and using the resulting  data to train a random subset of the networks. The idea is to use the variation in the learning to simulate some kind of confidence bound  so that the algorithm behaves a little bit like Thompson sampling.
The paper has some strengths and weaknesses. First of all, the experimental results are quite interesting, especially that the algorithm outperforms DQN on Atari. The results on the synthetic experiment are also interesting.  I have three main concerns about the paper.   1. There is significant difficulty in reconstructing what is precisely going on. For example, in Figure 1,  what exactly is a head? How many layers would it have? What is the "Frame"? I wish the paper would spend a lot more space explaining how exactly bootstrapped DQN operates (Appendix B cleared up a lot of my queries and I suggest this be moved into the main body).  2. The general approach involves partitioning (with some duplication) the samples between the heads with the idea that some heads will be optimistic and encouraging exploration. I think that's an interesting idea, but the setting where it is used is complicated. It would be useful if this was reduced to (say) a bandit setting without the neural network. The resulting algorithm will partition the data for each arm into K (possibly overlapping) sub-samples and use the empirical estimate from each partition at random in each step. This seems like it could be interesting, but I am worried that the partitioning will mean that a lot of data is essentially discarded when it comes to eliminating arms. Any thoughts on how much data efficiency is lost in simple settings? Can you prove regret guarantees in this setting?  3. The paper does an OK job at describing the experimental setup, but still it is complicated with a lot of engineering going on in the background. This presents two issues. First, it would take months to re-produce these experiments (besides the hardware requirements). Second, with such complicated algorithms it's hard to know what exactly is leading to the improvement. For this reason I find this kind of paper a little unscientific, but maybe this is how things have to be. I wonder, do the authors plan to release their code?  Overall I think this is an interesting idea, but the authors have not convinced me that this is a principled approach. The experimental results do look promising, however, and I'm sure there would be interest in this paper at NIPS. I wish the paper was more concrete, and also that code/data/network initialisation can be released. For me it is borderline.   Minor comments:  * L156-166: I can barely understand this paragraph, although I think I know what you want to say. First of all, there /are/ bandit algorithms that plan to explore. Notably the Gittins strategy, which treats the evolution of the posterior for each arm as a Markov chain. Besides this, the figure is  hard to understand. "Dashed lines indicate that the agent can plan ahead..." is too vague to be understood concretely.  * L176: What is $x$?  * L37: Might want to mention that these algorithms follow the sampled policy for awhile.  * L81: Please give more details. The state-space is finite? Continuous? What about the actions? In what space does theta lie? I can guess the answers to all these questions, but why not be precise?  * Can you say something about the computation required to implement the experiments? How long did the experiments take and on what kind of hardware?   * Just before Appendix D.2. "For training we used an epsilon-greedy ..." What does this mean exactly? You have epsilon-greedy exploration on top of the proposed strategy?


Professor Forcing: A New Algorithm for Training Recurrent Networks
The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-stepahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of the recurrent network to be the same when training the network and when sampling from the network over multiple time steps. We apply Professor Forcing to language modeling, vocal synthesis on raw waveforms, handwriting generation, and image generation. Empirically we find that Professor Forcing acts as a regularizer, improving test likelihood on character level Penn Treebank and sequential MNIST. We also find that the model qualitatively improves samples, especially when sampling for a large number of time steps. This is supported by human evaluation of sample quality. Trade-offs between Professor Forcing and Scheduled Sampling are discussed. We produce T-SNEs showing that Professor Forcing successfully makes the dynamics of the network during training and sampling more similar.

This paper proposes a method for training recurrent neural networks (RNN) in the framework of adversarial training. Since RNNs can be used to generate sequential data, the goal is to optimize the network parameters in such a way that the generated samples are hard to distinguish from real data. This is particularly interesting for RNNs as the classical training criterion only involves the prediction of the next symbol in the sequence. Given a sequence of symbols x_1, ..., x_t, the model is trained so as to output y_t as close to x_{t+1} as possible. Training that way does not provide models that are robust during generation, as a mistake at time t potentially makes the prediction at time t+k totally unreliable.  This idea is somewhat similar to the idea of computing a sentence-wide loss in the context of encode-decoder translation models. The loss can only be computed after a complete sequence has been generated.
The idea in this paper is interesting and well motivated. When training an RNN for generation, using the likelihood of the observed data is not the proper criterion.   Even though the problem and approach are interesting, I find the description of the training objective (Sec. 2) very confusing and unclear. Precise remarks follow: - The random variable y is present in two expectations in Eq. (1). - The function B takes y as an input, but actually does not need to. Given an RNN with a given sequence of inputs, all the y can be computed. Moreover, taking the expectation with respect to y following the distribution parametrized by \theta_g does not make sense. Also, I guess that the authors use an empirical approximation of this expectation. It might be cleared to write it as an average on training samples. - The adversarial training is a min-max problem (Googfellow et al., 2014). The authors state that they "perform SG steps on ..." and "always do gradient steps on ...". This means that the two parameters are optimized using different objective functions. One could probably get the same results by maximizing -log(D) instead of minimizing -log(1-D). This would allow the authors to write the optimization problem they try to solve. - The different terms of the objective are not weighted in any way. It would be interesting to see how a trade-off between NLL and the other terms plays in terms of final performance. If the adversarial objective is seen as a "regularizer" (l. 56), it would be good to use a weighting term.  The experimental evaluation regarding the distribution of hidden states and the t-SNE maps is a bit frustrating. No discussion in the paper is proposed and the only explanations are in the caption of Fig. 3. From what I understood, it simply shows the fact that hidden states diverge when generating sequences, as the input shifts away from the true data distribution. More discussion would be welcome.  As far as the experimental evaluation is concerned, the experiments could be more convincing. For the character-level language modeling, the paper compares models trained with and without the adversarial loss. An improvement potentially due to the regularizing effect is shown (0.02 Bits/character). My concern is that the baseline model is not trained properly. Other papers report much lower performances on Penn Treebank, see for instance a recent workshop paper at ICLR2016: Alternative Structures for Character-Level RNNs, Piotr Bojanowski, Armand Joulin and Tomas Mikolov (Table 1). The model they used was a plain RNN and it achieved a validation BPC of 1.42. When the baseline model is not properly tuned, it is hard to assess the impact of the regularization.  Overall this is an paper tackling an interesting and important problem. However, the presentation is a bit overly-complicated and unclear. Experiments exhibiting improved likelihood (imputed to regularization) on held-out data are not convincing and the third experimental claim (tSNE maps) is not properly discussed in the paper. Of course, we need to keep in mind that properly evaluating the performance of generative models is complicated, and the user study shows statistically significant improvements.


Robustness of classifiers: from adversarial to random noise
Several recent works have shown that state-of-the-art classifiers are vulnerable to worst-case (i.e., adversarial) perturbations of the datapoints. On the other hand, it has been empirically observed that these same classifiers are relatively robust to random noise. In this paper, we propose to study a semi-random noise regime that generalizes both the random and worst-case noise regimes. We propose the first quantitative analysis of the robustness of nonlinear classifiers in this general noise regime. We establish precise theoretical bounds on the robustness of classifiers in this general regime, which depend on the curvature of the classifier's decision boundary. Our bounds confirm and quantify the empirical observations that classifiers satisfying curvature constraints are robust to random noise. Moreover, we quantify the robustness of classifiers in terms of the subspace dimension in the semi-random noise regime, and show that our bounds remarkably interpolate between the worst-case and random noise regimes. We perform experiments and show that the derived bounds provide very accurate estimates when applied to various state-of-the-art deep neural networks and datasets. This result suggests bounds on the curvature of the classifiers' decision boundaries that we support experimentally, and more generally offers important insights onto the geometry of high dimensional classification problems.

This paper offers a thorough analysis of the effect of both worse-case (adversarial) and random noise in machine learning classifiers. It derives bounds that precisely describe the robustness of classifiers in function of the curvature of the decision boundary. This leads to some surprisingly (at least to me) general conclusions: * For random noise, the robustness of classifiers behaves as sqrt(d) times the distance from the datapoint to the classification boundary (where d denotes the dimension of the data) provided the curvature of the decision boundary is sufficiently small. This corroborates the intuition that random noise is less of an issue for high-dimensional data. On the other hand, how do we know the curvature of decision boundaries for general classifiers? * For semi-random noise, defined as the worst possible perturbation possible a random subspace of dimension m, the robustness behaves as sqrt(d/m) times the distance to the boundary. Hence, it is quite possible to find small perturbations that cause data misclassification, even within strong constraints.  These theoretical estimates are evaluated empirically on image recognition tasks using deep neural nets. Vice versa, it leads to some insight on the curvature of the decision boundary in deep neural nets.  Overall, this is very solid work that will likely appeal to the NIPS community. If I have one comments, it is that this work should be extended to many other types of algorithms to demonstrate its claimed generality.
This is very solid work and the paper is overall very pleasant to read. My one comment would be that this work seems to open up a lot of interesting future work when applied to other learners besides deep neural nets (and in some cases, we know something about the curvature of the decision boundaries (e.g. decision trees) that would make for interesting analysis. Are the authors interested in pursuing this?  Minor point: There is no reference in the paper to the proofs (in the supplementary materials). Make sure that they are easy to find in the final version.


Review Networks for Caption Generation
We propose a novel extension of the encoder-decoder framework, called a review network. The review network is generic and can enhance any existing encoderdecoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework improves over state-ofthe-art encoder-decoder systems on the tasks of image captioning and source code captioning.

The paper proposes a novel sequence-to-sequence architecture that includes an encoder and a decoder, as usual, but adds a "reviewer" module in the middle that can perform a fixed number of attentive iterations over the input representations in order to better represent the complete input sequence. This process can also be refined by adding an intermediate loss to guide it, for instance a bag-of-word loss over the target sequence. Experiments are conducted on two different tasks (image captioning and code captioning) and comparisons are made with other state-of-the-art approaches.
In the last few months, a few related approaches have been proposed that should probably be considered. I remember (at least): - Order Matters:..., by Vinyals et al, ICLR 2016 - Adaptive Computation..., by Graves, ArXiv 2016. The "Order Matters" paper proposes a "Process" module between the encoder and the decoder which has a lot of similarities with the "Attentive Input Reviewer" version I think. The "Adaptive Computation" paper bears similarities with the Decoder module I think.  I like the idea of the "discriminative supervision" which enable to re-use the same supervision in a different way (kind of bag-of-word supervision) but I would really like to see how important it is: it seems that in the experiment the \lambda factor that mediates between this loss and the usual loss is fixed, so we don't really know how important this is.  The experiments on COCO do not mention that on the MSCOCO website, more recent results are available and show better performance on about all metrics compared to the ones in the paper.  Another interesting experiment I wish was provided was the importance of T_r, which is set to 8 in the COCO experiment. What happens when this is changed (lower? higher? is there potential overfitting?).  Regarding the image captioning model using VGG, could you specify where was the input attention taken? (which layer of VGG)  Regarding the attentive input reviewer vs attentive output reviewer, I would have liked to have a discussion on when to use which and how they compare.  ========  I have read the author's response, which answered all my concerns. I think the updated paper should be accepted.