A Multiplicative Model for Learning Distributed Text-Based Attribute Representations
In this paper we propose a general framework for learning distributed representations of attributes: characteristics of text whose representations can be jointly learned with word embeddings. Attributes can correspond to a wide variety of concepts, such as document indicators (to learn sentence vectors), language indicators (to learn distributed language representations), meta-data and side information (such as the age, gender and industry of a blogger) or representations of authors. We describe a third-order model where word context and attribute vectors interact multiplicatively to predict the next word in a sequence. This leads to the notion of conditional word similarity: how meanings of words change when conditioned on different attributes. We perform several experimental tasks including sentiment classification, cross-lingual document classification, and blog authorship attribution. We also qualitatively evaluate conditional word neighbours and attribute-conditioned text generation.

This paper proposes to incorporate side information for improving vector-space embedding of words via an "attribute vector" that modulates the word-projection matrices. One could simply think of word-projection tensors (although, in practice the tensors are factorized) where the attribute vector provide the loadings for the tensor slices. This is studied in the context of log-bilinear language models, but the basic idea should be applicable to other word embedding work. The theory part of the paper is very well-written. However, it is in the experimental section that things get somewhat muddier. I would have preferred to see fewer experiments with greater depth than a barrage of experiments with little insight. Firstly, I am not sure what the experiments at the beginning of section 3 (contents of Tables 2 & 3) are really showing. Many models can generate text, and I am not sure these examples are showing anything in particular about the proposed model. Even a simple n-gram LM can interpolate between Bible and Caesar. Also, please mention what values of the initial learning rate, decay factor, momentum, and momentum increase factor were used. In section 3.1, what are the attributes? Is is just a sentence vector that is an average of all sub-phrases, or something else? Could you motivate why the choice is reasonable as an attribute? In section 3.2, please change the notation to make it clear that S is from l and S' and C_k are from l'. Perhaps v should really be v_l. Also, what are the attributes (x) in this case? In figure 1, right: you talk about a 1-hot attribute vector, but that's really the language-id vector and not the attribute vector. Also, do you have any insight as to why Germany and Deutschland are far apart while the translations of all other countries appear close together? Is it an error, some noise not accounted for, or showing something interesting about what the model learns (perhaps some us vs. them distinction, whereby Germany in English and Deutschland in German aren't learnt to be the same concept)?In section 3.3, if there are 234 unique attributes, why do you use a 100-dimensional attribute vector?


Learning Shuffle Ideals Under Restricted Distributions
The class of shuffle ideals is a fundamental sub-family of regular languages. The shuffle ideal generated by a string set U is the collection of all strings containing some string u ∈ U as a (not necessarily contiguous) subsequence. In spite of its apparent simplicity, the problem of learning a shuffle ideal from given data is known to be computationally intractable. In this paper, we study the PAC learnability of shuffle ideals and present positive results on this learning problem under element-wise independent and identical distributions and Markovian distributions in the statistical query model. A constrained generalization to learning shuffle ideals under product distributions is also provided. In the empirical direction, we propose a heuristic algorithm for learning shuffle ideals from given labeled strings under general unrestricted distributions. Experiments demonstrate the advantage for both efficiency and accuracy of our algorithm.

This paper studies the problem of learning shffle ideals in the PAC-learning model under some simple distributions such as product distribution, or distributions generated by Markov chains. A shuffle ideal is a particularly simple type of regular language. Given alphabet \Sigma, and a string u =(u_1, …, u_L) the prime shuffle ideal generated by u is the set of strings obtained by inserting any strings s_0, …, s_L in u to get s_0 u_1 s_1 … u_L s_L.  The shuffle ideal is a more general class of regular languages where instead of a string u we consider strings in a product set U = U_1 \times … \times U_L where U_i \subseteq \Sigma and take the union of the prime shuffle ideals generated by strings in U. The problem of learning regular languages has many applications. The theoretical study of this problem is in the PAC model. This however seems to be a hard problem, and so people try to study special cases and shuffle ideals are one such case. I think there’s a disconnect here between theory and practice as I don’t see any bridge between them mentioned in the paper under review or in [3]. However, I do find the problem being studied interesting in its own right; it seems natural and basic to me and thus could be of use in other contexts. The results are (seemingly) substantial generalization of previous work in that several of the results here are for learning shuffle ideals (as opposed to prime shuffle ideals)  and hold in situations where the distribution is generated by a Markov chain, or distribution over \Sigma^n where each position from 1 to n has its own distribution and one independently samples a symbol from \Sigma for each position. The algorithms work in the statistical query model and thus also in the PAC-learning model (for specific distributions). The paper also gives a simple greedy heuristic for learning prime shuffle ideals in the general case without proving any guarantees. Experimental results show it to be superior to the previous work. Thus the results seem to be interesting and non-trivial and represent progress. However, I have serious concerns regarding the technical parts as I now explain:[The following is based on the first 9 pages.]The writing is rather dry with little by way of proof outlines or intuitive explanation of the main ideas behind the proofs. The only explanation is after Theorem 3, which I didn't find very helpful. There should be a comparison with [3] in terms of new techniques and ideas. A cursory glance at [3] reveals high-level similarities in the notation and algorithms. So I suppose the present paper is building upon [3] and that makes it important to include a comparison with [3].The writing is not very clear; in most cases, where something is not clear I can guess what is meant but at some places I am not so sure. A few examples: The algorithm in Figure 3 seems to be missing some initialization. In particular, what’s V? What does it mean to approximately learn a shuffle ideal? I couldn’t find it being defined anywhere. Presumably it means that the symmetric difference between the learned concept and the actual concept is small? And when one is learning a prime shuffle ideal what does it mean? Is it that the Hamming distance or edit distance between the learned string and the actual string generating the ideal is small? But then that would require some quantification of approximation and the Theorems 1, 2, 3 do not have it.  What is \sigma in the statements of these theorems? Is it a parameter that one can choose or is it somehow related to approximation? Is L (as in Def. 2) a constant or logarithmic in n? I guess some small upper bound on L is needed to justify the use of conditioning like V \sqsubseteq x[1:n-1] in statistical query model. Also I’d expect L to show up in the statements of theorems, but it doesn’t.Mention that the problem is information-theoretically solvable by VC-dimension arguments (i.e. polynomial number of samples suffice though the algorithm may not be efficient). Thus I have to say that even the theorem statements are not completely clear to me. The paper would have been much better if the authors had devoted more space to one of the proofs (say Theorem 3), and given sufficient explanation to be convincing about its correctness. Other results could be stated in less detail to meet the page limit. This also makes me unable to comment on the correctness and novelty of the proofs, although I have no reason to have doubt.


Computing Nash Equilibria in Generalized Interdependent Security Games
We study the computational complexity of computing Nash equilibria in generalized interdependent-security (IDS) games. Like traditional IDS games, originally introduced by economists and risk-assessment experts Heal and Kunreuther about a decade ago, generalized IDS games model agents' voluntary investment decisions when facing potential direct risk and transfer-risk exposure from other agents. A distinct feature of generalized IDS games, however, is that full investment can reduce transfer risk. As a result, depending on the transfer-risk reduction level, generalized IDS games may exhibit strategic complementarity (SC) or strategic substitutability (SS). We consider three variants of generalized IDS games in which players exhibit only SC, only SS, and both SC+SS. We show that determining whether there is a pure-strategy Nash equilibrium (PSNE) in SC+SS-type games is NP-complete, while computing a single PSNE in SC-type games takes worst-case polynomial time. As for the problem of computing all mixedstrategy Nash equilibria (MSNE) efficiently, we produce a partial characterization. Whenever each agent in the game is indiscriminate in terms of the transfer-risk exposure to the other agents, a case that Kearns and Ortiz originally studied in the context of traditional IDS games in their NIPS 2003 paper, we can compute all MSNE that satisfy some ordering constraints in polynomial time in all three game variants. Yet, there is a computational barrier in the general (transfer) case: we show that the computational problem is as hard as the Pure-Nash-Extension problem, also originally introduced by Kearns and Ortiz, and that it is NP-complete for all three variants. Finally, we experimentally examine and discuss the practical impact that the additional protection from transfer risk allowed in generalized IDS games has on MSNE by solving several randomly-generated instances of SC+SS-type games with graph structures taken from several real-world datasets.

The paper introduces a new type of game, generalized interdependent security games, based on Kearns and Ortiz 2004. Several new complexity results are derived and proven for this class of games and some preliminary experimental investigation is given. This is a dense technical paper with many new (but quite specific) contributions for this new class of games. The problem is well-motivated, and the paper reads smoothly in sections 1 and 2. Section 3 (Computational results of alpha-IDS games) is difficult to follow. It feels like there is a lot, possibly too much, that the authors tried to fit into a single paper here and the main contribution/story gets a bit lost amidst the lemmas and proofs. The results are new and should be of interest to (at least) the same audience as that of [8], but the scope of the content does still feel a bit narrow. The experiments section is one long paragraph that lacks structure and polish. There is no conclusion nor much discussion of these results.Questions: Is \Delta_i^sc = \Delta_i^ss? Why two different terms, for clarity of the context? In proof of Theorem one, "its clause" and "its literal": why are these singular? Fig2 shows variables can belong to several clauses. Also, I cannot explain the bounds (1-q)^2 > \Delta_i^sc > (1-q)^3 and 1 > \Delta_i^ss > (1-q); these seems inconsistent with the text and best response definitions, is there an error here?Minor points: - your's -> yours- references, several missed capitalization (nash -> Nash)** Added after response. I appreciate the clarifications made by the authors and pointers to previous research. These games seem to have been more widely-studied than I was first led to believe after my first read of the paper. I encourage the authors to make this more clear if possible. I understand that space is a limiting factor, but the second half of the paper could certainly use some "smoothing", and especially some more discussion on these results could help the impact this work will have, if accepted.


Zeta Hull Pursuits: Learning Nonconvex Data Hulls
Selecting a small informative subset from a given dataset, also called column sampling, has drawn much attention in machine learning. For incorporating structured data information into column sampling, research efforts were devoted to the cases where data points are fitted with clusters, simplices, or general convex hulls. This paper aims to study nonconvex hull learning which has rarely been investigated in the literature. In order to learn data-adaptive nonconvex hulls, a novel approach is proposed based on a graph-theoretic measure that leverages graph cycles to characterize the structural complexities of input data points. Employing this measure, we present a greedy algorithmic framework, dubbed Zeta Hulls, to perform structured column sampling. The process of pursuing a Zeta hull involves the computation of matrix inverse. To accelerate the matrix inversion computation and reduce its space complexity as well, we exploit a low-rank approximation to the graph adjacency matrix by using an efficient anchor graph technique. Extensive experimental results show that data representation learned by Zeta Hulls can achieve state-of-the-art accuracy in text and image classification tasks.

"Zeta Hull Pursuits: Learning Non-convex Data Hulls" addresses the problem of column sampling in datasets by finding points that do not alter significantly the Ihara zeta function of a graph (presumably constructed via a kernel). They provide establish some basic facts about the zeta function and the effect that removing a vertex has on it.  These they use to provide a fast greedy algorithm for selecting 'outlying' vertices, and a faster version using anchors.  They demonstrate on some standard datasets that using these points can improve classification accuracy.    The paper is fairly well written, with good organization and only minor typos.  I have checked the proofs and did not find any mistakes.  I am not sure that the method is motivated enough to merit acceptance, while it is in fact a novel idea.The authors elude to the fact that this works for summarization of the data points without assuming that they are preserving the convex hull or enscribing ellipsoids.  I believe that this is the primary merit of this paper, that they are able to find representative points in a combinatorial fashion.  With that said, it is not exactly clear what is the motivation behind the use of the zeta function.  If they had clearly argued for the use of points that minimize their epsilon objective, perhaps via the learning of non-convex hulls (which was never clearly defined) then my review might be better.Also, I don't think that the Theorems are significant enough technical contributions to merit much attention.  Theorem 1 reiterates an established fact regarding the a determinant form of the Zeta function (generalized to weighted graphs).  Theorems 2 and 3 are straightforward results from matrix algebra and the SVD.


Learning with Fredholm Kernels
In this paper we propose a framework for supervised and semi-supervised learning based on reformulating the learning problem as a regularized Fredholm integral equation. Our approach fits naturally into the kernel framework and can be interpreted as constructing new data-dependent kernels, which we call Fredholm kernels. We proceed to discuss the "noise assumption" for semi-supervised learning and provide both theoretical and experimental evidence that Fredholm kernels can effectively utilize unlabeled data under the noise assumption. We demonstrate that methods based on Fredholm learning show very competitive performance in the standard semi-supervised learning setting.

This paper describes a framework particularly useful for semi-supervised learning based on Fredholm kernels.  The classical supervised learning optimization problem solved in kernel-based methods is extended to incorporate unlabeled information leading to discretized version of the Fredholm integral equation.QualityThe paper has high technical quality with well-supported claims by theoretical analysis and convincing experimental results.  The proposed formulation leads to a new data-dependent kernel that incorporates unlabeled information.  The classifying function differs from the classical represent theorem solution but it is still elegant and easy to compute.  I have few comments: One is  related to Equation 3 where the solution of the proposed optimization problem is described.  Since the Authors mentioned that the associated kernel need not be PSD, does Equation 3 still hold in this case ? A proof on how Equation 3 was obtained could have been nice to see too.  Another comment concerns the experimental section where the Authors chose an "optimal" parameter for all methods in the benchmark.  How were those parameters determined ? What is meant with "optimal" ?ClarityThe paper is clearly written and well-organized in most part.  It is easy to follow and the main ideas are explain adequately.  There are some typos though for example:* Line 82:  "... will a Euclidean ..." -> ... will be a Euclidean ..."* Line 83: "or" missing.* Section 4 title: "Fredhom" -> Fredholm* Line 247: "porportional" -> proportional* Line 283: "That is, or..." -> That is, for...* Line 352: Sentence needs to be rewritten.OriginalityThe proposed approach is related to a recent NIPS publication ([9]) but it is novel in essence.  It is clear how the work differs from this publication and others.  SignificanceThe results are important and significant.  There was an significant effort to test the proposed method on several datasets in different application domains.


PAC-Bayesian AUC classification and scoring
We develop a scoring and classification procedure based on the PAC-Bayesian approach and the AUC (Area Under Curve) criterion. We focus initially on the class of linear score functions. We derive PAC-Bayesian non-asymptotic bounds for two types of prior for the score parameters: a Gaussian prior, and a spike-and-slab prior; the latter makes it possible to perform feature selection. One important advantage of our approach is that it is amenable to powerful Bayesian computational tools. We derive in particular a Sequential Monte Carlo algorithm, as an efficient method which may be used as a gold standard, and an Expectation-Propagation algorithm, as a much faster but approximate method. We also extend our method to a class of non-linear score functions, essentially leading to a nonparametric procedure, by considering a Gaussian process prior.

--- Summary of the paperThis paper establishes PAC-Bayesian bounds with AUC risk, correspond to two different prior for linear classifiers.It also offers SMC and EP algorithms (the latter is recommended because of its light computational cost) designed for obtaining the pseudo-posterior distribution of classifier.-- QualityI did not follow all of mathematics in supplemental file, but the conditions, lemmas, and the derived bounds seem legitimate.Claims of the paper are partly supported by experiments, but the presentation of the experimental results should be improved. Description of the datasets and explanation of experimental results are not very precise. Also, somewhat sloppy mathematical expressions slightly impairs the quality of the paper.-- ClarityThe clarity of the paper should be improved. Some mathematical symbols are used without definition, and some notations lack consistency. For example, "S" in theorem 2.3 and 2.4, and "\Xi" in subsection 3.1. The main contributions I think of this paper are establishing PAC-Bayesian bounds with two different priors, and implementing EP algorithm for PAC-Bayesian approach.Since description of experimental setting and discussion on the results are not enough, I think Section 4 can be removed, just mentioning that the results can be extended to non-linear case, to make room for further explanation and discussion on the results.For example, it is better to include the number of samples of each dataset to Table 1. Also, I think the "Glass" dataset has more than two classes. I'd like to know how the authors cast the dataset to binary-labelled dataset.Labels on axes of plots in all of the figures should be larger. They are so small that it is barely readable.From Figure 3(a), I cannot observe the absorbing effects claimed in the paper. Also, there should be more explanation on Figure 3(b).-- Originality and significanceI have never read papers deriving PAC-Bayesian bound for AUC risk, and hence the use of EP for estimating the Gibbs posterior with AUC risk is new to me.The derivation of bounds and algorithm seems sound, and they agree with previously known results.Since the proposed algorithm works well, particularly for highly unbalanced problems, the paper conveys not only theoretically novel finding but also provides a practically useful algorithm.


Near-optimal sample compression for nearest neighbors
We present the first sample compression algorithm for nearest neighbors with nontrivial performance guarantees. We complement these guarantees by demonstrating almost matching hardness lower bounds, which show that our bound is nearly optimal. Our result yields new insight into margin-based nearest neighbor classification in metric spaces and allows us to significantly sharpen and simplify existing bounds. Some encouraging empirical results are also presented.

This paper discusses the problem of compressing a set of examples for nearest neighbor (NN) classification. Namely given a set of examples S in some metric space the goal is to output a subset S' of S such that nearest-neighbor labeling of points from S given examples from S' gives the correct labels for all points in S.It is first observed that if points are separated by at least \gamma then any \gamma-net gives a desired set. Together with standard results for doubling dimension this implies an upper bound of (1/\gamma)^ddim(S) on the size of S'. The authors discuss a greedy algorithm for constructing a \gamma-net and also ways to speed it up and further reduce the number of points heuristically.In the second result, that is the main technical result they show that the problem is NP-hard to approximate to within a factor of  2^{(\log (1/\gamma)ddim(S))^{1-o(1)}}. NP-hardness of the problem was known but this is a much stronger inapproximability result. It is obtained via an interesting  reduction from a label cover.Finally the authors demonstrate the performance of 1-NN after their sample compression algorithms on several data sets and compare it with using NN directly. The results show that the algorithm does not affect the error of the hypothesis. This is somewhat strange since Occam razor would suggest that there should be perceptible improvement. In either case smaller sample also speeds up NN classification so applying the compression could be useful.I think that sample compression for NN classification is a natural aspect of this fundamental ML technique and this work makes an interesting contribution to understanding of the problem. Having a nearly optimal lower bound is particularly nice. The techniques are combinations of standard tools and there are no particular surprises here. The paper is also well-written.Some additional comments:line 139: agnostic learning is defined for a class of functions. What you describe is just a basic setup for binary classification from iid examples.line 240: I do not see a definition of c (the weight)Sec 2: it would be useful to discuss whether the problem remains hard in the Euclidean spaceline 362: you suggest that ddim(X) is larger than ddim(S). This does not seem clear to me.


Content-based recommendations with Poisson factorization
We develop collaborative topic Poisson factorization (CTPF), a generative model of articles and reader preferences. CTPF can be used to build recommender systems by learning from reader histories and content to recommend personalized articles of interest. In detail, CTPF models both reader behavior and article texts with Poisson distributions, connecting the latent topics that represent the texts with the latent preferences that represent the readers. This provides better recommendations than competing methods and gives an interpretable latent space for understanding patterns of readership. Further, we exploit stochastic variational inference to model massive real-world datasets. For example, we can fit CPTF to the full arXiv usage dataset, which contains over 43 million ratings and 42 million word counts, within a day. We demonstrate empirically that our model outperforms several baselines, including the previous state-of-the art approach.

The authors propose a new topic model collaborative topic Poisson factorization (CTPF) which combines ideas from Poisson factorization (PF) and collaborative topic regression (CTR). The idea behind the generative model is intuitive and appealing: learn a latent representation for users and articles that explains the observed ratings as well as the content of the articles. The authors use stochastic variational inference which enables them to apply the model to large datasets. The authors claim that the use of a shared latent representation allows them to attack the cold-start recommendation problem. I like the idea a lot and I think it can potentially be very useful (e.g. as a web service for discovering new scientific articles) assuming the authors' claims are true. However, the presentation needs to be improved before I can whole-heartedly accept the paper.- I felt that the paper is targeted at someone who knows the PF and CTR; the text in some sections reads like a "diff file" highlighting differences between the CTPF model from the older models :) It'd be better to present the CTPF model first and then perhaps describe how PF and CTR arise as special cases of CTPF. - Figure 1: It might be better to move it to the discussion section. \theta and \epsilon are not defined in section 1. I was confused by what "topic offsets" meant until I read the latter sections. (Line 72 describes the red bars as "preferences of readers who have the EM paper in their libraries", which is not helpful either.)- Evaluation: Do you assume that articles not rated are irrelevant? If so, does it not add noise to the computed precision and recall? Are these precision/recall values significant enough that users of such a system would notice an improvement in quality. Can you provide the performance of the "ideal" recommender which sorts by true relevance? Perhaps measures such as Normalized Discounted Cumulative Gain (NDCG) used in information retrieval would be more useful?- The evaluation compares models just based on precision and recall. I think it would be better to compare other model properties as well (e.g. distribution over number of ratings per article, distribution of number of ratings per article)- Line 88: Not clear to me why Poisson factorization handles sparse data better than Gaussian factorization. It's very common in recommender systems to let the objective function depend just on the observed ratings which requires us to update just the latent vectors for users/articles with non-zero observations.- Line 130: PF model seems to use a separate rate parameter (gamma distribution) for each document and user. Is a separate rate not needed here? Line 140: \epsilon is referred to as "document user weights" here and "topic offsets" elsewhere, which makes it hard to follow. It'd be better to use a consistent reference. - How was K set in the experiments? How sensitive is the algorithm to Gamma hyper-parameters? Some sensitivity analysis on toy data might be helpful.Line 374: "Content [1]" is misleading. The fLDA paper uses different priors, likelihood and a different inference scheme. I would encourage the authors to compare to fLDA if the source code is available. I guess "topic offset variables" refers to \epsilon?- Line 414: why are train/test/validation proportions different for the two datasets? 1% of documents as a test set seems too small. Do you average results on multiple train/test splits? - Figure 5: what do you mean by "readers in the field"? In A), B), C) it would be simpler if you specify what you mean in terms of \theta and \epsilon. There's some discrepancy between the figure and the caption ("ontologies and applications" vs "information retrieval").- Line 431: The paper seems to end quite abruptly. It'd be good to have a discussion section to summarize the contributions and discuss future directions.Typos, etc:- Line 107: use r instead of y- Line 285: \tau_0, \kappa not defined. Update (after author rebuttal):The authors have addressed most of my concerns. I have updated my rating, trusting the authors to make the necessary changes.I think it'd be helpful to add a brief discussion early on about implicit feedback and how it differs from explicit feedback, for example the differences explained in Hu et al. [13]. IMO, the PF model is more appealing in the implicit feedback setting; the authors' response about why PF is better than competitors in the implicit feedback setting might be worth adding to the main text. The fact that unobserved responses are treated as 0s in the evaluation is a bit unsatisfying and I am slightly worried how this assumption affects the authors' claims about 4x improvement over state-of-the-art. That said, I understand that evaluation on implicit feedback datasets is less straightforward. I strongly encourage the authors to make the data as well as the code publicly available so that others can study this problem in more detail.


Conditional Swap Regret and Conditional Correlated Equilibrium
We introduce a natural extension of the notion of swap regret, conditional swap regret, that allows for action modifications conditioned on the player's action history. We prove a series of new results for conditional swap regret minimization. We present algorithms for minimizing conditional swap regret with bounded conditioning history. We further extend these results to the case where conditional swaps are considered only for a subset of actions. We also define a new notion of equilibrium, conditional correlated equilibrium, that is tightly connected to the notion of conditional swap regret: when all players follow conditional swap regret minimization strategies, then the empirical distribution approaches this equilibrium. Finally, we extend our results to the multi-armed bandit scenario.

This paper discusses a type of swap regret which is conditioned on past actions of the user. They connected this concept with a new type of correlated equilibria. They designed hardness results for having the behavior conditional on the whole past, and came up with an algorithm which minimized the new type of regret. Finally, it showed how to deal with this type of regret in a bandit setting.First of all, I think that this is an interesting area of research. However, I didn't see what I look for in new concepts of equilibrium, namely whether one can cooperate with one's opponent and still minimize regret. Secondly, I found that the paper was unclear when it could have been done so much more cleanly. Specifically, instead of a convoluted description of interleaving multiple regret algorithms, they could have simply started from the algorithms based upon regret matching and swap regret.Specifically, you can break down the regret into "regret on time T from playing action i after action j instead of action k".r^T_{i,j,k}=p^T_i p^{T-1}_j (f^T_i - f^T_k)Then, define the total regret as:R^T_{i,j,k}=\sum_{t=2}^T r^t_{i,j,k}Then, bigram swap regret is:Regret_T=\sum_{i} \max_{k} R^T_{i,j,k}Thus, if all of the R^T_{i,j,k} are increasing sublinearly, you're good. Define:R^{T,+}_{i,j,k}=max(0, R^T_{i,j,k})Blackwell's approachability theorem needs:\sum_{i,j,k} r^{T+1}_{i,j,k} R^{T,+}_{i,j,k} \leq 0which implies:\sum_{i,j,k} p^{T+1}_i p^{T}_j (f^{T+1}_i - f^{T+1}_k) R^{T,+}_{i,j,k} \leq 0We know p^T and R^{T,+}, so we pull that to the right:\sum_{i,k} p^{T+1}_i (f^{T+1}_i - f^{T+1}_k) \sum_j p^{T}_j R^{T,+}_{i,j,k} \leq 0You can define a value:R^{T,+}_{i,k} = \sum_j p^{T}_j R^{T,+}_{i,j,k}So, our new desired inequality is:\sum_{i,k} p^{T+1}_i (f^{T+1}_i - f^{T+1}_k) R^{T,+}_{i,k} \leq 0This is exactly swap regret. Moreover, it extends to n-grams easily (r^T_{i,j,k,l}=p^T_i p^{T_1}_j p^{T-2}_k (f_l - f_i) for trigams). If you don't care about a particular transition in a particular state (as in state-dependent bounds), you can zero out that R^T_{i,j,k}.T_{i,j,k}. So, this seems like a much simpler way to do what they are trying to do.


Exploiting easy data in online optimization
We consider the problem of online optimization, where a learner chooses a decision from a given decision set and suffers some loss associated with the decision and the state of the environment. The learner's objective is to minimize its cumulative regret against the best fixed decision in hindsight. Over the past few decades numerous variants have been considered, with many algorithms designed to achieve sub-linear regret in the worst case. However, this level of robustness comes at a cost. Proposed algorithms are often over-conservative, failing to adapt to the actual complexity of the loss sequence which is often far from the worst case. In this paper we introduce a general algorithm that, provided with a "safe" learning algorithm and an opportunistic "benchmark", can effectively combine good worst-case guarantees with much improved performance on "easy" data. We derive general theoretical bounds on the regret of the proposed algorithm and discuss its implementation in a wide range of applications, notably in the problem of learning with shifting experts (a recent COLT open problem). Finally, we provide numerical simulations in the setting of prediction with expert advice with comparisons to the state of the art.

This work presents a new way to exploit 'easy' data in online convexoptimization and in the bandit setting. Given a strategy A that has goodworst-case guarantees, and another opportunistic strategy B that might,for some data, get much smaller regret, it is shown how to combine A andB into a single strategy that gets the best of both worlds, at the costof adding O(sqrt(T*log(T))) to the regret of A and a small constant tothe regret of B. The same approach is shown to work for prediction withexpert advice, online convex optimization and a relaxation of the banditsetting in which *two* arms can be observed in every round.This is pretty cool, because previous authors have needed to usecomplicated special-purpose strategies that depended on A and B toachieve similar results.Applications are given that 1) take B to be Follow-the-Leader and getperformance similar to the FlipFlop algorithm of De Rooij et al. [8]; 2)address a recent open problem about tracking the best expert, and 3)adapt to worst-case and stochastic bandits as in the work of Bubeck andSlivkins [5]. In these applications, the results of prior work are notexactly recovered, but the authors generally provide a good discussionof the differences.The paper makes me wonder about a natural follow-up question: Is itpossible to generalize the approach to include more than oneopportunistic benchmark strategy B and get small regret with respect toall of them? Currently the additive term O(sqrt(T*log(T))) preventsnesting the approach multiple levels.Some remarks:Tracking the best expert:  - the proposed method needs to know the gap delta between the best    expert in each interval and the second-best, which is unknown in    practice. I would suggest adding a comment that mentions this.The comparison to FlipFlop on p.5:  - This comparison focusses on using specific parameters for FlipFlop.    These parameters trade off the performance in the worst-case with    the overhead with respect to FTL. So perhaps a fairer comparison    could be made by setting them to make the worst-case for FlipFlop    and for (A,B)-Prod the same. Alternatively, the discussion might be    simplified by not focussing on particular parameter values too much.  - The comment about competing with the actual regret of    AdaHedge vs its regret bound seems misplaced, because the difference    between the two is negligible compared to the additional    sqrt(T*log(T)) term. line 267: why does D-Prod guarantee regret of O(T^{1/4}) when FTL and    AdaHedge both have regret Theta(sqrt(T))? Suppose we generate losses    for 2 experts from the uniform distribution (as in the lower bound    from "How to use expert advice"), then any algorithm will have    regret Theta(sqrt{T}), so how can this be true? Which algorithm is    used as the benchmark in this case?Minor comments:In figure 2, w_{1,A} is not really a parameter, because it is fixed to  be 1-w_{1,B}.In Thm 1, why introduce mu_A and mu_B, when they are the same as  w_{1,A} and w_{1,B}?In Corollary 1, I think your result requires C >= e^4 or something.line 245: "a factor of" -> "an additive term of" (factors are multiplicative)Proposition 1: say that there is a proof in the appendixIn the discussion of your result for the bandit setting, you could also  mention the recent paper "One Practical Algorithm for Both Stochastic  and Adversarial Bandits" by Seldin and Slivkins at ICML 2014.line 488: there is a minus missing: -log(1-eta)/eta is increasingline 497: doesn't this hold with equality?


Multivariate Regression with Calibration ⇤
We propose a new method named calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models. Compared to existing methods, CMR calibrates the regularization for each regression task with respect to its noise level so that it is simultaneously tuning insensitive and achieves an improved finite-sample performance. Computationally, we develop an efficient smoothed proximal gradient algorithm which has a worst-case iteration complexity O(1/✏), where ✏ is a pre-specified numerical accuracy. Theoretically, we prove that CMR achieves the optimal rate of convergence in parameter estimation. We illustrate the usefulness of CMR by thorough numerical simulations and show that CMR consistently outperforms other high dimensional multivariate regression methods. We also apply CMR on a brain activity prediction problem and find that CMR is as competitive as the handcrafted model created by human experts.

This paper proposed a new method called calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models. The assumption is that the noise matrix has an uncorrelated structure, and different regression tasks have different noise variances. Instead of using the same tuning parameter lambda for all the regression tasks, CMR calibrates different tasks by solving a penalized weighted least square problem weights define in (2.3) (i.e., some kind of estimate of noise standard deviation). The paper is well-structured and clearly-written. The appendix contains a lot of technical details. The idea of CMR formulation is technically sound. The proposed computational algorithms and demonstrated statistical properties make sense at a high level, although I did not check every step of the derivation and proof given the limited expertise in this field. The empirical study on simulated and real data yielded promising results.Line 129: Should "weighted least square program" read "weighted least square problem"?Another straightforward approach to calibrate the parameter lambda is to run multiple regression on each response variable separately. How does this compare with the proposed CMR? It would be interesting to compare them on both simulated and real data. Sometimes simple models might yield better or more interpretable results on real data than more advanced ones.


Conditional Random Field Autoencoders for Unsupervised Structured Prediction
We introduce a framework for unsupervised learning of structured predictors with overlapping, global features. Each input's latent representation is predicted conditional on the observed data using a feature-rich conditional random field (CRF). Then a reconstruction of the input is (re)generated, conditional on the latent structure, using a generative model which factorizes similarly to the CRF. The autoencoder formulation enables efficient exact inference without resorting to unrealistic independence assumptions or restricting the kinds of features that can be used. We illustrate connections to traditional autoencoders, posterior regularization, and multi-view learning. We then show competitive results with instantiations of the framework for two canonical tasks in natural language processing: part-of-speech induction and bitext word alignment, and show that training the proposed model can be substantially more efficient than a comparable feature-rich baseline.

REPLY TO AUTHOR RESPONSEThanks for the response.  You write: "It is possible we missed a related paper in the neural network literature. We would be grateful if you could provide citations of neural architectures which used hand-crafted features for unsupervised learning of structured outputs."  This may have misunderstood my point.  I was really just making the same point as reviewer_26 (you responded to him or her).  But as it happens, I know of one branch of work loosely related to your question.  Stoyanov et al. (AISTATS 2011, NAACL 2012) showed how to derive the structure of a sum-product feed-forward network by unrolling loopy sum-product belief propagation on a given graphical model.  Each potential function in the graphical model can be parameterized in terms of hand-crafted features of its incident variables (e.g., it may be log-linear).  These potentials convert into parametric weights of the network, reflecting the hand-crafted features.  In addition, the structure of the network reflects the structure of the graphical model.  The network is then trained by backprop to minimize prediction error at designated output variables.  Is this "unsupervised"?  Well, like the original graphical model, the resulting feed-forward network learns to predict the latent variables of the graphical model from the input variables in a way that is helpful for predicting the output variables.  This could be regarded as "unsupervised" learning of a structure over the latent variables given the (input, output) variables.  If the graphical model were given an auto-encoder structure where output = input, then their method would result in an unsupervised system very much like yours.SUMMARYUnsupervised learning of latent structure y given input x.  This is a bottleneck method using an autoencoder p(x | y) p(y | x).  Specifically, p(y | x) is a domain-specific CRF, while p(x | y) appears to be a simple independent model \prod_i p(x_i | y_i).  The  y are compared with human annotations.It is efficient to compute the objective (2) and its gradient.  The objective is regularized log-loss of the reconstruction, marginalizing over y.  More generally, the autoencoder is p(\hat{x} | y,phi) p(y | \hat{x},x,phi) where \hat{x} is some ad hoc simplified version of x and phi is side information.  COMMENTSThe results are quite good.  In particular, the method seems to produce fairly stable improvements across multiple languages (8 for POS tagging, 3 for alignment as evaluated by MT).  However, the paper and supplementary material appear to have been written hastily.  It is often hard to figure out the details of what was done experimentally. The method is formally attractive and might turn out to be an important building block in learning certain kinds of latent structure.  (Particularly if reconstructing the input x is only one task in a multi-task learning setting, so that y should also be useful to predict other, supervised properties of the input x.  The authors don't discuss this.)However, the authors don't do a good job of explaining why the method will do a good job of unsupervised learning.  Their objective is to match human annotations, not just to learn representations that are useful for some other task.  The usual difficulty in this setting is that the latent variables end up learning the "wrong" properties of the data -- not the properties that the humans annotated.  The authors say something about this problem at L067, but they don't really explain why their method should do any better.  Nor do they do any analysis of the experimental results to understand what was learned and why.  There are single-number comparisons with other methods, but no breakdowns or targeted experiments aimed at understanding what is going on.  It's just left as a happy mystery.  (There are no learning curves, either.)  It would also be helpful to explore whether the improved results are achieved because they reduce model error or reduce search error.  For example, suppose method A (previous) and method B (this paper) find latent distributions p_A and p_B over the latent variables y.  If each method is initialized separately at p_A and p_B, then does method A's objective prefer p_A and method B's objective prefer p_B, which implies that the objective is better?  Or do they both prefer p_A or both prefer p_B, which suggests that the different stems from a search bias?The authors suggest (L262) that their autoencoder architecture is more appropriate in some respects than neural autoencoders, although this assertion is not defended by discussion, empirical comparison, or error analysis.  I think the authors are mainly motivated by the fact that the p(y | x) model can build on a long line of efficient, feature-engineered work on supervised structured prediction work in NLP.  But there are many architectures that could use such features, including neural architectures.DETAILED COMMENTS TO AUTHORThe p(\hat{x} | y) model is described only obliquely.  I have a good guess what it is, but please spell it out!What exactly are you evaluating?  At L172 you mention the distribution p(y | x,\hat{x},phi), under which, presumably, the probability of a given latent y is proportional to p(\hat{x} | y,phi) p(y | x,phi).  Does this mean that you are evaluating the 1-best y from this distribution?  One or many samples from this distrubtion?  Something else?It's not possible to really understand the features.  For example, supplement L103 says that auto+full features include functions of the y_i, but supplement Table 2 doesn't show what those features are.  It is also not clear whether "x_i, x_{i-1}" in the "full" column of that table is talking about conjoined features or is a list of features.  And it's not clear whether the full model also includes all of the h&k features or not.L147: Even before you introduced phi, you already allowed side information for the encoding phase, namely anything in x that was not part of \hat{x}.  So perhaps you want to say that phi is side information that is (also) available in the reconstruction phase?Possibly relevant is http://www.cs.cmu.edu/~nasmith/papers/gimpel+smith.naacl12b.pdf .Suggestion on how to handle the supplementary material: Currently this is formatted as a separate paper, which is confusing.  Please set it up as appendices to the main paper, following the bibliography, without duplicate material.  This will allow you to have a single bibliography, a single set of figure and equation numbers, and crossrefs with the main paper.  After running through pdflatex, use pdftk to divide the document into the main paper and the supplement.


Scale Adaptive Blind Deblurring
The presence of noise and small scale structures usually leads to large kernel estimation errors in blind image deblurring empirically, if not a total failure. We present a scale space perspective on blind deblurring algorithms, and introduce a cascaded scale space formulation for blind deblurring. This new formulation suggests a natural approach robust to noise and small scale structures through tying the estimation across multiple scales and balancing the contributions of different scales automatically by learning from data. The proposed formulation also allows to handle non-uniform blur with a straightforward extension. Experiments are conducted on both benchmark dataset and real-world images to validate the effectiveness of the proposed method. One surprising finding based on our approach is that blur kernel estimation is not necessarily best at the finest scale.

The paper starts from the hypothesis that image structures that are “small scale” or with gradient profiles distributed near-each other are invariably recovered incorrectly when the size of the blur kernel is similar to their size. A consequence of this is poor kernel estimatesintentionally blurring as done in a coarse-to-fine deblurring process naturally suppresses the contributions of these small scale structures since their contributions is downplayed at coarser levels.Building on these ideas, the paper suggests a joint blink deblurring across the different scales with the constraint that the latent image remains the same for all scales.Parameters are tuned/learnt to account for different levels of noise Pros- Novel idea of jointly deblurring across multiple scales with constraints.- Improvement over competing methods.Cons- Claims of newer analysis not supported or overstated (?)- Details on non-uniform blur models missing1) The idea of jointly deblurring across multi-scales, to the best of my knowledge, is new.it raises some question though.First, wouldn’t such a method over penalize errors in the coarser scales — since they are present in all the scales ? Second, the paper suggests the finer scales are less robust to noise and while coarser scales are robust to noise, they lack fine detail.While i agree with this motivation, does the proposed method make better use of it than other techniques — say Fergus et al. where a coarse to fine refinement is performed under the same motivation.2) The paper keeps suggesting that while similar techniques have been proposed, the  paper provides motivation and analysis.I am not sure what analysis the paper refers to.There are experimental results that suggests that the  joint multi-scale deblurring has benefits. A lot of claims are made in lines 262-294. But these are largely empirical and many details are missing. What was the sample size with respect to number of latent images, number of blur kernels,  and hence, wouldnt quite pass for a rigorous analysis.3) The paper could do a better job in describing the non-uniform model since they are used in the real examples. Specifically, reporting how the homographs were generated and the number used etc. should help reproducibility of the results.Smaller comments- As opposed to “Estimation error” which is very hard to interpret, the paper should report scores with SNR or PSNR. There are normalized and in log-scale — both of which make it significantly easier to interpret.- Line 283. “This corroborated previous findings” — citations are required here.- Figure 6. Pl show recovered blur kernels for all images. These are valuable in interpreting the results.


Provable Tensor Factorization with Missing Data
We study the problem of low-rank tensor factorization in the presence of missing data. We ask the following question: how many sampled entries do we need, to efficiently and exactly reconstruct a tensor with a low-rank orthogonal decomposition? We propose a novel alternating minimization based method which iteratively refines estimates of the singular vectors. We show that under certain standard assumptions, our method can recover a three-mode n × n × n dimensional rank-r tensor exactly from O(n 3/2 r 5 log 4 n) randomly sampled entries. In the process of proving this result, we solve two challenging sub-problems for tensors with missing data. First, in analyzing the initialization step, we prove a generalization of a celebrated result by Szemerédie et al. on the spectrum of random graphs. We show that this initialization step alone is sufficient to achieve the root mean squared error on the parameters bounded by C(r 2 n 3/2 (log n) 4 /|Ω|) from |Ω| observed entries for some constant C independent of n and r. Next, we prove global convergence of alternating minimization with this good initialization. Simulations suggest that the dependence of the sample size on the dimensionality n is indeed tight.

This paper proposes an algorithm for the exact recovery of low-rank tensors from randomly chosen O(r^5n^{3/2}log^4(n)) entries (in the case of rank-r symmetric n x n x n tensor).The proposed algorithm uses the robust tensor power method of Anandkumar et al. [1] to obtain an inital solution for the least squares iteration, which is a popular approach for tensor factorization. The proof relies on a rather strong notion of incoherence of the factor matrix and the proposed algorithm relies on the knowledge of the correct rank and the incoherence parameter.The theoretically predicted scaling is confirmed in a synthetic experiment except that the authors heuristically assume a milder dependency with respect to the rank r (r^{1/2} instead of r^5).The strength of the paper is that as far as I know, this is the first paper that presents a tensor completion algorithm with a O(n^{3/2}) guarantee from randomly chosen entries, though the authors have missed two related studies: Krishnamurthy & Singh (NIPS 2013) have shown a O(n) sample complexity when the samples are adaptively chosen; Tomioka et al. (NIPS 2011) have analyzed the random design setting and have shown a O(n^2) sample complexity.Another interesting point is that it uses the robust tensor power method as an initalizer for a more conventional iterative algorithm, which have shown to be an effective strategy by other studies (Chaganty & Liang, ICML 2013).On the negative side, the incoherence assumption employed by the authors seems rather strong. It basically ensures that all the entries of the true tensor T have the magnitude of O(n^{-3/2}). Compared to the fact that it is possible to show a tight sample complexity when the entries have magnitude O(1) in matrix completion (Foygel & Srebro, COLT2011), this is rather disappointing.The algorithm also assumes the knowledge of the correct rank r and the incoherence parameter mu. It was not clear to me what happens if these parameters were mis-specified.Minor comments:- sigma_max and sigima_min whould be defined when they first appear (in page 3).- I am not sure if the claim "the above theorem implies that among n^2 fibers of the form ... it is sufficient to have only O(n^{3/2}) fibers with any samples" (page 4) is true. At least it is not implied by the theorem because the theorem assumes uniform sampling. For example if only the first entries for all 3n^2 fibers were sampled, would this be enough to recover any incoherent low-rank tensor? This is not true for r > 1 for matrix completion because removing any observed entry would make the graph disconnected (see Kiraly and Tomioka, ICML 2012).- The claim that the degrees of freedom of a symmetric 3rd order tensor is rn-r^2 (page 4) needs a proof or a reference.


How transferable are features in deep neural networks?
Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

This paper systematically studies the transferrability of deep convnet features across a few experimental settings.      The experiments are focused on the "AlexNet" architecture and the ILSVRC-2012 dataset, with the main set of results on transfer learning problems induced by dividing the 1000 ILSVRC-2012 object classes into random 500/500 splits.      The N-layer network is trained on split A, then layers 1 through k are copied to a new network and layers k+1 through N are randomly initialized and retrained on split B.      Results are reported across both choices of allowing layers 1 through k to be fine-tuned vs. frozen, and across all choices of k.      Another set of results shows these in the setting of a manually chosen split of natural vs. man-made classes, and yet another set of results explores using random, un-trained weights in some layers.      Many interesting observations are made which were not previously reported, particularly results #3 and #5 in section 4.1.      Despite the large number of papers that either extract features from or fine-tune the ImageNet-pretrained "AlexNet" CNN, this is the first work I know of to do a clean, thorough exploration of the space of choices one has to make when using this class of methods.      While the paper offers little in the way of technical or mathematical novelty, it does not strive to -- it provides an extensive, clean set of numerical answers to many commonly asked questions, and I believe its results give valuable practical insight to researchers and practitioners of modern deep learning methods.      The authors also promise to release the code they use to run these experiments, allowing other researchers to explore how these experiments generalize to other architectures, etc.


Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimization
Consider the problem of minimizing the sum of a smooth (possibly non-convex) and a convex (possibly nonsmooth) function involving a large number of variables. A popular approach to solve this problem is the block coordinate descent (BCD) method whereby at each iteration only one variable block is updated while the remaining variables are held fixed. With the recent advances in the developments of the multi-core parallel processing technology, it is desirable to parallelize the BCD method by allowing multiple blocks to be updated simultaneously at each iteration of the algorithm. In this work, we propose an inexact parallel BCD approach where at each iteration, a subset of the variables is updated in parallel by minimizing convex approximations of the original objective function. We investigate the convergence of this parallel BCD method for both randomized and cyclic variable selection rules. We analyze the asymptotic and non-asymptotic convergence behavior of the algorithm for both convex and non-convex objective functions. The numerical experiments suggest that for a special case of Lasso minimization problem, the cyclic block selection rule can outperform the randomized rule.

***** Update after author feedback ******I agree with the authors' response and trust them to make the suggested changes that will improve the paper. On the other hand, as this added discussion won't be double checked by reviewers, I strongly encourage them to have the modification reviewed by a critical colleague in the field to ensure its quality.*******************quality:  7 (out of 10)clarity: 8originality: 6significance: 7SUMMARY: The authors propose a simple parallel scheme for the block-coordinate optimization of the sum of a smooth (possibly non-convex) function with a non-smooth convex block-separable function. The scheme finds a target point for each block in parallel in a chosen subset of blocks by minimizing the sum of a strongly convex approximation to the smooth part on this block (with matching gradients) and the non-smooth part. Each block in this subset is then updated (in parallel) as a convex combination of the previous value and the target points. A parallel proximal gradient scheme can be obtained as a special case; though using a convex combination of the iterates yield a slightly different scheme than previous work. The suggested algorithm is very similar to [9], except that in [9] the subset was chosen using a greedy scheme (which can be expensive), whereas this submission explores both randomized schemes or a cyclic scheme. For these, the authors prove the asymptotic convergence to a stationary point of the algorithm under standard Lipschitz gradient conditions. They also prove a O(1/k) convergence rate (where k is the number of cycles for the cyclic scheme) to the global minimum for the convex case or for the square of the norm of the objective proximal gradient  for the non-convex case, for the algorithm with a constant step-size. They provide a succinct experimental comparison for a Lasso problem, observing that the cyclic scheme performs better than the randomized one, and that a speed-up can be obtained through parallelization (up to a point where the communication overhead starts to dominate).EVALUATION:I think this paper makes an interesting contribution to NIPS. There has been a lot of interest in parallel optimization schemes lately, and the one proposed in this paper, by analyzing general convex approximations to non-convex smooth objectives, is fairly general. Pieces of this submission has all been done somewhere else, but the combination is novel. For example, successive convex approximation schemes for non-convex problems was already analyzed in [24], but not for parallel updates. The contributions over [9] is to analyze the cyclic and randomized subset selection scheme, as well as to provide global convergence bounds (rather than just asymptotic convergence as in [9]). A contribution over several other parallel block coordinate  schemes which was proposed recently is to consider the general successive convex approximation approach (instead of standard proximal gradient updates), which can give tighter bounds.The paper is easy to follow, and the proofs are fairly self-contained. I was disappointed though that the authors didn't provide the proof for the cyclic updates for Lemma 3 (line 313-314) in supplementary material.The numerical experiments are mainly a sanity check, and don't necessarily provide that much insights. The proof of asymptotic convergence (p. 5 mainly, as p.4 is used in other proofs) don't provide much insights and is fairly technical -- I would have suggested to put this part in an appendix in the supplementary material, and use the extra page freed to actually discuss more the meaning of their convergence rate results and how they compare with all the related schemes proposed in the literature. For example, how do the constants compare; what kind of speed-up is expected for different standard scenarios; what insights do the convergence analysis provide for practical usage of their algorithm (especially for a NIPS audience); etc. I think this would make the paper more useful to the NIPS community.Pros:- Simple and general parallel optimization scheme for smooth (possibly non-convex) objective with non-smooth separable part. The generality of the setup could be useful for the types of problems considered by the NIPS community.- Fairly self-contained (and easy to read) convergence analysis for both convex and non-convex case, including both cyclic and randomized subset schemes.Cons:- No discussion providing insights about their convergence results or how they compare with other results in the literature.- Toy empirical comparison.== Other comments ==Typos:- Line 167, \tau_min is not defined.- Line 253: the limit should be removed in this sentence, i.e. "... we have h(...) < = h( ...) for all x_i ..., *for all j*.


On Multiplicative Multitask Feature Learning
We investigate a general framework of multiplicative multitask feature learning which decomposes each task's model parameters into a multiplication of two components. One of the components is used across all tasks and the other component is task-specific. Several previous methods have been proposed as special cases of our framework. We study the theoretical properties of this framework when different regularization conditions are applied to the two decomposed components. We prove that this framework is mathematically equivalent to the widely used multitask feature learning methods that are based on a joint regularization of all model parameters, but with a more general form of regularizers. Further, an analytical formula is derived for the across-task component as related to the taskspecific component for all these regularizers, leading to a better understanding of the shrinkage effect. Study of this framework motivates new multitask learning algorithms. We propose two new learning formulations by varying the parameters in the proposed framework. Empirical studies have revealed the relative advantages of the two new formulations by comparing with the state of the art, which provides instructive insights into the feature learning problem with multiple tasks.

UPDATE:After reading the feedback, I now understand that Lemma 1 should be correct. Since the proof also confuses the other reviewer, I suggest the authors to improve the presentation of Lemma 1 to avoid such confusion. Also, may the paper accepted, please emphasize the impact of the equivalence.  =========== This paper studies a particular type of multi-task learning, in which the models are decomposed into the product of two components. The authors have established these class of models with group Lasso models. This idea is really interesting and gives many insights into the two sets of formulations. The authors extend the models based on these insights with new parameters and designed an alternating optimization algorithm to solve the problem. The theoretical contribution of the paper is interesting. However, I have some concerns on this paper. First of all, the Lemma does not seem to be right. In Lemma1, to satisfy the equality of Cauchy-Schwarz, we need a specific relationship between sigma_j and alpha. However, in (3), we note that both alpha and sigma are variables to be optimized, and in the optimal solution of (3), it is not necessary the case that this relationship holds, and thus the equality of Cauchy-Schwarz may not hold. As such, the causality here does not seem correct to me.If authors can show that Lemma 1 is correct, then the equivalence of the two types of formulations is novel. Besides the novelty, the authors should further elaborate on why the equivalence is important. The value of such equivalence should be the insights given by it. In this paper, since this equivalence does not have much implication on the efficiency (one is an efficient alternative to the other in some scenario), the new formulations derived based on the equivalence should be the most important part to convince reader its importance. Unfortunately, the paper does not say much about the new formulations, neither does it provide a comprehensive empirical study on them.  Why p and k are chosen to be only 1 or 2? Is there any justification on choices? If there is no theoretical justification, it is also interesting to provide empirical observations on different p, k combinations other than fixing them to be 1 and 2 (e.g., fraction numbers). In the Table 2, we see that in the real datasets, the proposed two formulations MMTFL typically cannot improve much from the existing approaches. This give me concern of how really useful are these newly proposed methods. I would encourage the authors to explore on more real multi-task learning datasets and try to explore what are those datasets that the proposed method works well and why this is the case.


Constant Nullspace Strong Convexity and Fast Convergence of Proximal Methods under High-Dimensional Settings
State of the art statistical estimators for high-dimensional problems take the form of regularized, and hence non-smooth, convex programs. A key facet of these statistical estimation problems is that these are typically not strongly convex under a high-dimensional sampling regime when the Hessian matrix becomes rankdeficient. Under vanilla convexity however, proximal optimization methods attain only a sublinear rate. In this paper, we investigate a novel variant of strong convexity, which we call Constant Nullspace Strong Convexity (CNSC), where we require that the objective function be strongly convex only over a constant subspace. As we show, the CNSC condition is naturally satisfied by high-dimensional statistical estimators. We then analyze the behavior of proximal methods under this CNSC condition: we show global linear convergence of Proximal Gradient and local quadratic convergence of Proximal Newton Method, when the regularization function comprising the statistical estimator is decomposable. We corroborate our theory via numerical experiments, and show a qualitative difference in the convergence rates of the proximal algorithms when the loss function does satisfy the CNSC condition.

This paper studies constant nullspace strong convexity, strongly convex only over a constant subspace, that can result in a linear convergence rate for the proximal gradient method and quadratic convergence rate for the proximal Newton method, even if the objective function is not strongly convex on the whole space. This work is an important extension of recent convergence rate analysis on sparse and low-rank minimization problems, where the objective functions are usually l1 norm or nuclear norm and the linear operators in the constraint usually have some nullspace property. The authors also extend the analysis to general decomposable norms.However, the paper has the following issues:1. The proximal Newton method does not seem practical in reality. The subproblem (10) is usually not easily solvable, unless h(x) is a quadratic function. That all subproblems should be easily solvable is a basic requirement on a practical algorithm. Otherwise, may I simply assume that (1) is easily solvable, whatever it is? Indeed, the authors have used randomized coordinate descent to solve (10), which hardly has any guarantee on accuracy of the solution to (10). So the subsequent quadratic convergence does not make sense at all. This algorithm is also awkward as it requires an extra linear search. And the authors do not specify when to stop line search (only wrote "sufficient decrease"). So I don't think analysis on the proximal Newton method adds much contribution.2. The authors overclaim the contribution a bit. Actually, this paper only analyzes the case of h(x) being a decomposable norm, not a general convex function. So the title, abstract and introduction should all be fixed.3. There may be some errors in the proofs that need clarification from the authors:a. In (22), it is unclear why the first inequality holds.b. In the third line below (22), it is unclear why the l2-norms of a_j and \bar{a}_j are both 1. Note that in the definition decomposable norm, the dual norms of a_j and \bar{a}_j are both 1.4. In the second line below (29), "\epsilon" and "L\epsilon" should switch. In experiments, where is 'x' in the loss functions? And why the minimal objective function values are both zeros? Finally, the authors assume that f(x) is twice continuous differentiable but I don't think that the hinge loss satisfies this condition.5. In Fig. 1, I don't think that it is obvious that the upper curve corresponds to linear convergence. And the convergence is surprisingly slow -- at the order of millions of iterations! Similarly, in Fig. 2 that the upper curve is in quadratic convergence is not obvious either.


Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm
We improve a recent guarantee of Bach and Moulines on the linear convergence of SGD for smooth and strongly convex objectives, reducing a quadratic dependence on the strong convexity to a linear dependence. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence on average smoothness, dominating previous results, and more broadly discus how importance sampling for SGD can improve convergence also in other scenarios. Our results are based on a connection between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods.

This paper considers stochastic gradient descent, for the problem of minimizing F(x) = \E[f_i(x)]. The paper considers several variants of the problem, depending on whether the functions f_i are smooth, strongly convex, Lipschitz continuous gradient, or just Lipschitz continuous. We discuss these individually more below.In the introduction, the main point of the paper seems to be the improvement of the rate for linear convergence. Indeed, in the special setting where the optimizer of F is also the optimizer of each individual setting (the authors call this the “realizable setting” which is not very applicable in most problems in machine learning, save for a set of linear equations that are all satisfiable) there is linear convergence. But the paper is nice and has a consistent theme even for less exciting convergence regimes. The authors might consider outlining this more in the introduction, since before I got to the second half of the paper, my impression was that most of the results concerned this very limited regime.In the first short part of the paper, the authors show that by using a co-coercivity lemma (that gives a bound on the squared difference of the gradient, based on the Lipschitz constant) the dependence in the optimality guarantees in Bach-Moulines on the square of the Lipschitz constants, can be reduced to dependence on the supremum of the Lipschitz constant. While this is a simple result to obtain, using a known result (the co-coercivity lemma) it is a nice observation, and is the starting point for the rest of the paper. In particular, it shows that if the supremum of the Lipschitz constants is not much different than their average, then the dependence on the average Lipschitz constant is reduced from the square appearing in Bach-Moulines, to linear dependence. The supremum, of course, could be significantly bigger than the average. But the main point of this result is that the authors then show that (a) the dependence is real and not a shortcoming of the analysis (it cannot be improved), and that (b) this dependence on sup L_i indicates a natural algorithmic manner of correcting/improving it: importance sampling. Accordingly, the rest of the paper is about this modification: rather than sample the gradients uniformly, one can sample them according to a different distribution. The authors consider several settings here, including extensions to the non-smooth case, non-strongly convex case, and also the most common For the smooth and strongly convex case, in the realizable setting, the authors show that the optimal choice is to twist with a distribution proportional to the ratio of Lipschitz constant of the individual function, to the average Lipschitz constant, which is essentially what one would expect from the failure to avoid dependency on the supremum value in the ordinary SGD setting. When, as in any regression setting with noise, the problem is not realizable, this choice comes at a cost. The authors then show that one gets the best of both worlds by doing an averaged sampling, weighting by a mixture of the distribution mentioned above, and the uniform distribution. Using this scheme, the modified SGD algorithm dominates ordinary SGD (and the analysis in Bach-Moulines). Using essentially similar analysis, the authors also consider the non-smooth case, the non-strongly convex case, and the non-realizable case, in each showing that the modified “importance sampling” SGD gives some improvements. Perhaps the only catch here is that doing this modified sampling may come at a cost (at the very least a cost of sample complexity) depending on how we have access to the samples we draw. There is a short discussion here. Is there something more to say about the additional cost?Overall, this is a nice paper. The initial focus on the realizable case doesn’t seem to be too compelling. I don’t know of too many cases when we have overdetermined systems where every single equality can be simultaneously satisfied. But there is plenty of interesting material in the rest of the paper.The paper is generally well written. There are a few very minor typos I found:46: select —> selects91: seems like it should be epsilon_0/epsilon instead of the other way around.119: we the —> we use the226: get a —> we get a264: a reduction of in the number ofpp 7 2nd paragraph: then —> than330: weighing —> weighting383: rewighting —> reweighting


Distributed Power-law Graph Computing: Theoretical and Empirical Analysis
With the emergence of big graphs in a variety of real applications like social networks, machine learning based on distributed graph-computing (DGC) frameworks has attracted much attention from big data machine learning community. In DGC frameworks, the graph partitioning (GP) strategy plays a key role to affect the performance, including the workload balance and communication cost. Typically, the degree distributions of natural graphs from real applications follow skewed power laws, which makes GP a challenging task. Recently, many methods have been proposed to solve the GP problem. However, the existing GP methods cannot achieve satisfactory performance for applications with power-law graphs. In this paper, we propose a novel vertex-cut method, called degree-based hashing (DBH), for GP. DBH makes effective use of the skewed degree distributions for GP. We theoretically prove that DBH can achieve lower communication cost than existing methods and can simultaneously guarantee good workload balance. Furthermore, empirical results on several large power-law graphs also show that DBH can outperform the state of the art.

1. The authors suggest a method for vertex-cut-based graph partitioning for distributed computation. Targeting good performance on naturally occurring power-law graphs, they introduce a hash-based method that works (in its simplest form) as follows:Every edge e is uniquely assigned to a machine by selecting the incident vertex with the smallest degree and hashing that vertex's id. This intuitively leads to a balanced partitioning. The authors further provide analysis showing that their method performs at least as good as currently used graph partitioning schemes. Finally, experimental evaluation shows that for the task of PageRank, their method actually outperforms the state of the art.Pros:The idea is simple and clear. The solution is intuitive and apparently effective. The figures contribute to readability. Simple analysis guarantees that the method is definitely not a bad idea and the experiments show strict improvement over the state of the art.The authors have performed a number of experiments, however sticking to PageRank exclusively.The suggest GP method pertains to the very timely topic of graph-computation, it is very easy to implement and it is reasonable to assume that it will have impact on the community.Cons:The biggest issue with the paper is organization and readability. While there are crisp ideas in the material, the presentation is lacking. Ideas do not flow well and there are some grammatical errors. Should the paper be accepted it would need a lot of editing.The idea itself is not the big contribution; it feels like a very reasonable approach. It’s the experimental evaluation and, secondarily the analysis, that give this paper merit.


PEWA: Patch-based Exponentially Weighted Aggregation for image denoising
Patch-based methods have been widely used for noise reduction in recent years. In this paper, we propose a general statistical aggregation method which combines image patches denoised with several commonly-used algorithms. We show that weakly denoised versions of the input image obtained with standard methods, can serve to compute an efficient patch-based aggregated estimator. In our approach, we evaluate the Stein's Unbiased Risk Estimator (SURE) of each denoised candidate image patch and use this information to compute the exponential weighted aggregation (EWA) estimator. The aggregation method is flexible enough to combine any standard denoising algorithm and has an interpretation with Gibbs distribution. The denoising algorithm (PEWA) is based on a MCMC sampling and is able to produce results that are comparable to the current state-of-the-art.

This paper proposes a general aggregation method for image denoising by combining multiple weakly denoised images from standard methods. The proposed method is based on EWA and it employs Monte-Carlo simulation for efficient approximation. The results are comparable to the state-of-the-art algorithms.It is nice for the authors to provide a thorough review on the image denoising literature and to discusses the difference of PEWA with existing denoising algorithms in the beginning. As far as I understand, the main contributions of this paper (Sec 4) compared to other "internal" methods are that PEWA generate candidate patches from weakly denoised images and that it merges candidate patches based on EWA. The aggregation method is flexible enough to combine any standard denoising algorithm and has a nice interpretation with Gibbs distribution. The experiments show that the aggregated image can be significantly better than each component, and good results can be obtained efficiently.I have a concern on the flexibility of the proposed method. The authors criticize the "external" methods for not being flexible for unknown noise level in line 55-57. But PEWA also relies on a known noise level. What if the noise variance is unknown? And what if the noise is not iid Gaussian?I'm also worried about the fact that when combining a large number of denoised images, the quality will drop (line 244-246). Would that suggest the SURE is not good enough to combine a large number of patches, or the MCMC approach is not mixing well?In the experiment section, PEWA is shown to perform just similar to and sometimes worse than the state-of-the-art. It would be useful to discuss any advantage of PEWA compared to BM3D and NL-Bayes.


Active Regression by Stratification
We propose a new active learning algorithm for parametric linear regression with random design. We provide finite sample convergence guarantees for general distributions in the misspecified model. This is the first active learner for this setting that provably can improve over passive learning. Unlike other learning settings (such as classification), in regression the passive learning rate of O(1/ ) cannot in general be improved upon. Nonetheless, the so-called 'constant' in the rate of convergence, which is characterized by a distribution-dependent risk, can be improved in many cases. For a given distribution, achieving the optimal risk requires prior knowledge of the distribution. Following the stratification technique advocated in Monte-Carlo function integration, our active learner approaches the optimal risk using piecewise constant approximations.

This work studies the problem of active learning in linear regression.Unlike classification, the objective in active regression is merely to improve the constant factors in the distribution-dependent rates of convergence compared to passive learning, since it is known that the asymptotic dependence on the number of labels typically cannot be improved compared to passive learning, and nor can the worst-casevalues of the constant factor.  The paper argues that there is a distribution-dependent constant factor in the rate of convergence of passive learning, which can sometimes be improved for active learning.Specifically, they propose a rejection sampling scheme, which altersthe sampling distribution to a more-favorable one, without changingthe optimal solution.  However, the rejection sampler requires a sort of scaling function \phi as a parameter, and obtaining good performance guarantees requires one to set this function carefully, and with some dependence on the joint distribution of (X,Y).  Since this latter distribution is unknown, the algorithm attempts to optimize the choice of \phi amongpiecewise constant functions, using an estimated linear function from an initial sample.  They prove a risk bound for this method, which can be made to approach the ``oracle'' rate (where the optimal \phi is given),and which is provably sometimes superior to the capabilities of passive learning methods.Overall, this seems to be a solid contribution, which I suspect will have a wide audience.My one main reservation is that I would have liked to see more discussionof the dependence on K vs \rho*_A in Theorem 5.1.  There are some termsthat are increasing in K, while we would like \rho*_A to decrease toward\rho*, which presumably requires K to grow.  Thus, the trade-off betweenthese two quantities can affect the rates.  Some examples to illustrate how we should expect this trade-off to behave, for some reasonable distributions and sensible partition, would be helpful.minor comments: One citation that is missing here is Efromovich (2005): Sequential Designand Estimation in Heteroscedastic Nonparametric Regression.That work studies active regression as well (though for a nonparametric class),and also finds improvements in constant factors based on the degree of heteroscedasticity.I also want to echo a remark of one of the other reviewers, that the notation for P_\phi seems a little strange to me.  I believe the intention is that, denoting by Q_\phi the measure having density \phi with respect to D, we define P_\phi as the distribution of (X/\sqrt{\phi(X)},Y/\sqrt{\phi(X)}), for (X,Y) \sim Q_\phi. This P_\phi should then be well-defined, and L(w,D)=L(w,P_\phi) would then follow via the law of the unconscious statistician.Was this the intended meaning of P_\phi?


Local Linear Convergence of Forward-Backward under Partial Smoothness
In this paper, we consider the Forward-Backward proximal splitting algorithm to minimize the sum of two proper closed convex functions, one of which having a Lipschitz continuous gradient and the other being partly smooth relative to an active manifold M. We propose a generic framework under which we show that the Forward-Backward (i) correctly identifies the active manifold M in a finite number of iterations, and then (ii) enters a local linear convergence regime that we characterize precisely. This gives a grounded and unified explanation to the typical behaviour that has been observed numerically for many problems encompassed in our framework, including the Lasso, the group Lasso, the fused Lasso and the nuclear norm regularization to name a few. These results may have numerous applications including in signal/image processing processing, sparse recovery and machine learning.

The FBS algorithm has been rediscovered in the last 10 years along with other splitting algorithms like the Douglas-Rachford algorithm. FBS is a simple algorithm to minimize a functional which consists of the sum of a smooth function and one for which the prox-operator can be solved efficiently. It first applies a gradient descent step (thus it is a first-order method) on the smooth part after which the prox-operator of the nonsmooth part is applied. This algorithm is used in many machine learning and imaging applications and a lot of research has been done to speed it up (FISTA, Barzilai-Borwein step sizes etc.).Hence, analyzing the convergence properties of FBS which is the topic of the current paper is interesting for the NIPS community.The main contribution of this paper is to make use of the concept of partial smoothness to derive convergence results given in Theorem 3.1. Although this idea was employed before, the results seem novel and interesting. Generally, I think that the paper would have improved a lot if the authors put the convergence proof in the supplementary material and spend these two pages to discuss their results in more detail. For example, the results in Remark 3.2 are interesting but it should be made clear whether they can be found somewhere, are easy to obtain or their proof is too long to be included here. In the last sentence of the abstract, the authors mention the importance for applications but only Remark 3.4 follows this thought. A natural question is how one can make use of the finite identification property, i.e., if there is a way to identify the index K of part 2. of the proof and how the algorithm can be adapted once K is reached. One could at least experimentally analyze finite identification property. The plots provided here focus on the asymptotic behavior. The example problems considered in the paper are well-chosen since one clearly sees differences according to cases i) and ii) of the main theorem. However, showing maybe one real-world example would be beneficial. In addition, especially for readers new to the topic, it would be worthwhile to make the step from Theorem 3.1 to the examples easier by explicitly checking all conditions of the theorem, even ones which might be clear like (3.1). The last paragraph on p. 5 about the difference in the tightness of the convergence rate are very interesting and should be treated in broader detail. Is there a mathematical justification why exactly the bottom two convergence curves are less tight? (apart from the general observation on the properties of the manifold and the sign vectors)Furthermore, an outlook on generalizations (if possible) of the results presented here would be good: Do the results generalize to an infinite dimensional setting? Are they transferable to other splitting methods like the Douglas-Rachford method? Can parts of the results be carried over to the non-convex setting? How about versions of FBS which employ speed-up strategies, e.g. FISTA?Minor things:- define polyhedral gauge functions- p. 3, l. 128: define "generalized sign vector"- p. 3, l. 127: typo: subsclass - p. 5, l. 234 typo: taut string algorithm


Bayesian Nonlinear Support Vector Machines and Discriminative Factor Modeling
A new Bayesian formulation is developed for nonlinear support vector machines (SVMs), based on a Gaussian process and with the SVM hinge loss expressed as a scaled mixture of normals. We then integrate the Bayesian SVM into a factor model, in which feature learning and nonlinear classifier design are performed jointly; almost all previous work on such discriminative feature learning has assumed a linear classifier. Inference is performed with expectation conditional maximization (ECM) and Markov Chain Monte Carlo (MCMC). An extensive set of experiments demonstrate the utility of using a nonlinear Bayesian SVM within discriminative feature learning and factor modeling, from the standpoints of accuracy and interpretability.

This paper mainly extends the Bayesian linear SVM in [5] to a nonlinear version, and further combines the Bayesian nonlinear SVM with factor models.The extension from linear to nonlinear however is quite trivial by simple adopting the standard kernel tricks. The resulted nonlinear version involves more complicated inference problem since it will also learn the kernel function parameters.The combination with factor models is produced by taking the two objectives together, while kernels are produced on the factor representations.There is not much novelity in terms of model extension and combination strategies. The overall learning problem is in fact a quite complicated non-convex optimization problem.Under the probabilistic Bayesian framework, some inference procedures are introduced to perform learning but there is no analysis about the complexity of the overall inference procedure.The experiments are limited to using Gaussian kernels. Is it possible to use other types of nonlinear kernels? Will it affect the inference algorithm?The datasets used in the experiments are too small (see Table 1). Large scale experiments need to be conducted.Moreover, the authors only compared the proposed approach to SVM and GPC methods.Considering the tasks addressed in this paper are simple binary classification tasks, why not compare to more advanced state-of-art methods?The authors motivate the work from the perspective of discriminative feature-learning models, which is a very general topic.I do not feel related works on this topic have been sufficiently discussed in the related work section.