Flexible Models for Microclustering with Application to Entity Resolution
Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points. Finite mixture models, Dirichlet process mixture models, and Pitman-Yor process mixture models make this assumption, as do all other infinitely exchangeable clustering models. However, for some applications, this assumption is inappropriate. For example, when performing entity resolution, the size of each cluster should be unrelated to the size of the data set, and each cluster should contain a negligible fraction of the total number of data points. These applications require models that yield clusters whose sizes grow sublinearly with the size of the data set. We address this requirement by defining the microclustering property and introducing a new class of models that can exhibit this property. We compare models within this class to two commonly used clustering models using four entity-resolution data sets.

The paper proposes a distribution over partitions of integers that supports micro-clustering. Specifically, the paper shows that that for existing infinitely exchangeable clustering models, such as Dirichlet Process (DP) and Pitman-Yor Process (PYP) mixture models, the cluster sizes grow linearly with the number of data points N. For applications such as entity resolution, the paper defines the micro-clustering property, where the ratio of the cluster size and number of data points N goes to 0 as N goes to infinity. The paper proposes a general distribution framework over partitions of integers that satisfies this micro-clustering property. This is done by first sampling the number of clusters from a distribution with positive integers as support, and then explicitly sampling each of the cluster sizes from a distribution over cluster sizes again with positive integers as support. This model achieves the micro-clustering property by sacrificing consistency of marginals, while preserving exchangeability. The paper then proposes two specific instances of this framework. The first uses the negative binomial for both distributions. The second uses a Dirichlet with an infinite dimensional base distribution for the distribution over cluster sizes to provide more flexibility for large datasets. Reseating algorithms similar to the Chinese Restaurant Process and the Pitman-Yor Process are provided for both models. Making use of the exchangeability property, sampling based algorithms are used for inference in both models. Experiments over 4 semi-synthetic datasets are used to illustrate that the proposed models outperform models without the micro-clustering property (DP and PYP) for the entity resolution task.
The following are the main strengths of the paper. + It points out and defines an important property of cluster sizes that existing infinitely exchangeable clustering models do not satisfy. There could be many applications, including and not limited to entity resolution, that require this property to be satisfied. + It proposes a framework for defining infinitely exchangeable clustering models that satisfy this micro-clustering property, and analyzes why the DP mixture model is an unsatisfactory instance of this class. It then proposes two specific and interesting instances of this class using specific distributions for the number of clusters and cluster sizes and derives reseating algorithms for these instances. + Detailed experimental results over semi-synthetic data illustrate usefulness of the proposed models to some extent for the entity resolution task. Experiments in the supplement show that draws from the proposed model satisfy the micro-clustering property. (This should be moved to the main paper.)  But the paper also has certain deficiencies, some of which are probably fixable. - The paper points out towards the beginning that it preserves exchangeability and sacrifices consistency of the marginal distributions to achieve the micro-clustering property. While I can see the benefits of preserving exchangeability (e.g. in designing sampling based inference algorithms), the price of sacrificing consistency is not clear. The paper has a vague statement in line 109: "It does not produce an exact sample if used to incrementally construct a partition". What does this mean? Is this related to the consistency issue? - While the paper has empirical comparisons with the Pitman-Yor Process, there is no theoretical analysis of the differences in the reseating algorithms. I can understand that the goal is to have significantly heavier tails in the cluster size distribution compared to the DP / CRP. The PYP addresses this by making the probability of picking a new table proportional to the current number of clusters. The reseating algorithm for both proposed models looks similar. Which aspect of the proposed reseating helps it to satisfy the micro-clustering property while the PYP does not? - The paper also does not satisfactorily analyze the relationship between the two proposed models. The Negative Binomial in the first model is replaced by a Dirichlet with an infinite dimensional base distribution in the second. In the second case, I could think of the cluster size probabilities as drawn from a GEM / Poisson-Dirichlet with two parameters. Would this also become too restricted? Or is the difference due to two different priors over infinite dimensional multinomial parameters, e.g. Negative Binomial and GEM?  - Early on, the paper mentions related work (the Uniform Process) that achieves the same goal differently: by sacrificing exchangeability instead of consistency. This work is never mentioned later in the paper. Some discussion of this earlier paper is necessary to understand the contributions of the proposed framework. - The experiments are not convincing enough for appreciating the usefulness of the proposed models for the entity resolution task. Judging by the error rates for all the models, the task for the first two datasets seem too simple for these datasets to be significant. For the other two datasets (which the paper claims are too hard), the two baseline models seem to be doing better.  Some less important comments: - Where is alpha in the reseating algorithm in lines 131-133? - If the authors feel that the Chaperones algorithm is significant, they need to describe it at least at a high level in the main paper. Mentioning it just by name in the main paper is not helpful. - The flow of the paper could be changed for better understanding. Lines 112-115 and 134-139 could be moved to the inference section, and lines 116-118 and 140-144 could be moved to the experiments section.  - The authors should reconsider the name of the proposed model. They should at least change it to the Flexible Micro-clustering Model instead of the Flexible Model for Micro-clustering model.  - In Eqn 3 and line 85, |C|_N should be |C_N|. In line 84, cluster |c| should be cluster c. - Eqn 5 should have a,q in the conditional. Eqn 7 should have a,q,alpha in the conditional. - What does "K is one observation" in line 122 mean? - In Eqn 11, please explain that theta_lk is the lth feature for the kth partition. - (The partition features of) One synthetic dataset can be imagined as one draw from (the entity features of) the corresponding real dataset. Are experimental results reported over a single random synthetic dataset or averaged over multiple samples? Though the noise parameters may be set to achieve low noise in expectation, it is possible to get high noise in a single random draw.  - What is the difference between the two Syria datasets?  - How are the error rates defined accounting for the label permutation issue? Are these defined over pairs of data points? - Some sentences contain unnecessary repetitions, e.g. in lines 150 and 214-216


Deep Alternative Neural Network: Exploring Contexts as Early as Possible for Action Recognition
Contexts are crucial for action recognition in video. Current methods often mine contexts after extracting hierarchical local features and focus on their high-order encodings. This paper instead explores contexts as early as possible and leverages their evolutions for action recognition. In particular, we introduce a novel architecture called deep alternative neural network (DANN) stacking alternative layers. Each alternative layer consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as local feature learner while the latter is used to collect contexts. Compared with feed-forward neural networks, DANN learns contexts of local features from the very beginning. This setting helps to preserve hierarchical context evolutions which we show are essential to recognize similar actions. Besides, we present an adaptive method to determine the temporal size for network input based on optical flow energy, and develop a volumetric pyramid pooling layer to deal with input clips of arbitrary sizes. We demonstrate the advantages of DANN on two benchmarks HMDB51 and UCF101 and report competitive or superior results to the state-of-the-art.

The authors argue that the utilizing context early in an deep neural  network architecture for action recognition in videos is beneficial. To this  end, they propose a novel architecture called deep alternative neural network (DANN)  which can utilize context as early as possible. DANN uses volumetric convolution  layer in conjunction with a recurrent layer to incorporate context into the  architecture. They empirically  demonstrate the benefits of their proposed model on standard datasets, and  compare their performance against other state of the art methods. In addition to  exploring the contexts early in the architecture, their proposed model is also  capable of handling arbitrary sized inputs through the usage of a volumetric  pyramid pooling layer. They also provide a mechanism  for adaptive fragmentation of a given video that can be beneficial for improved  performance.
Detailed review:  The problem of action recognition is well established in the literature. The  authors provide an interesting deep neural network based approach for  leveraging the context early on in the network architecture. They empirically show  the advantage of exploring context as early as possible in the deep  architecture. They also demonstrate  that exploring contexts at different levels of the architecture is also beneficial.  The capability of handling arbitrary sized input is also useful.   Novelty :  Their usage of context early in the network seems novel. The capability  to use arbitrary length input is also helpful.   Clarity :  The explanations seem sound.   Technical Quality:  The authors seem to provide the necessary technical details.  They also provide implementation details about  the actual experiments performed by them.   However, there are some questions/doubts/comments which were brought up by  the manuscript.  1) In the lower equation which is part of the Equation set 2, what is u_{(i-1)j}^{xyz} on the RHS. Is it the output of the previous layer (which can theoretically  be a recurrent layer unrolled T times )?  2) In Figure 3, is the right side dotted box the expansion of output->output  cycle in the left side dotted box, where the cycle repeats T times ?  3) An example that explains the author's comments on line 97-98 would have been  useful.  4) Some clarification regarding line 110 would be useful.  5) Explanation regarding the claim presented in line 144-145 would be nice.  What causes 3 times speedup when 2 GPUs are used instead of 1. that using 2 GPUs can provide 3 times speedup as compare to 1 GPU ?  6) At what layers are the dropouts used ?  7) How many random video clips are selected per video during data augmentation ?  8) What is the benefit of using a sliding window approach during testing , while  using random clips during training? If the aim is to get video level score,  then can the testing also be done using fixed sized random clips.


Multi-step learning and underlying structure in statistical models
In multi-step learning, where a final learning task is accomplished via a sequence of intermediate learning tasks, the intuition is that successive steps or levels transform the initial data into representations more and more "suited" to the final learning task. A related principle arises in transfer-learning where Baxter (2000) proposed a theoretical framework to study how learning multiple tasks transforms the inductive bias of a learner. The most widespread multi-step learning approach is semisupervised learning with two steps: unsupervised, then supervised. Several authors (Castelli-Cover, 1996;Balcan-Blum, 2005; Niyogi, 2008;Ben-David et al, 2008;Urner et al, 2011) have analyzed SSL, with Balcan-Blum (2005) proposing a version of the PAC learning framework augmented by a "compatibility function" to link concept class and unlabeled data distribution. We propose to analyze SSL and other multi-step learning approaches, much in the spirit of Baxter's framework, by defining a learning problem generatively as a joint statistical model on X ⇥ Y . This determines in a natural way the class of conditional distributions that are possible with each marginal, and amounts to an abstract form of compatibility function. It also allows to analyze both discrete and non-discrete settings. As tool for our analysis, we define a notion of -uniform shattering for statistical models. We use this to give conditions on the marginal and conditional models which imply an advantage for multi-step learning approaches. In particular, we recover a more general version of a result of Poggio et al (2012): under mild hypotheses a multi-step approach which learns features invariant under successive factors of a finite group of invariances has sample complexity requirements that are additive rather than multiplicative in the size of the subgroups.

The paper studies prediction problems (a binary realizable, separable classification) within a standard PAC framework and tries to quantifies benefits which a learning algorithm can achieve by using a certain structure of the unknown data-generating distribution $P(X,Y)$. More precisely, the authors consider problems where (a) the conditional probability $P(Y=1|X=x)$ function depends on the input $x$ only through a sufficient statistic (or the *representation*) $t(x)$ and (b) function $t(.)$ depends on the unknown distribution $P(X,Y)$ only through its marginal $P(X)$.   The authors approach this problem by providing  (1) the upper bound on the generalization error of the "oracle learner" having access to the structure and  (2) the lower bound for the performance of a standard learner with no knowledge regarding the structure in $P(X,Y)$ (Theorem 3).  The lower bound is based on a novel extension of a fat-shattering dimension (Definition 1), which measures the complexity of a set of *distributions*, while the upper bound trivially follows from the standard worst-case bounds on the ERM algorithm.   Finally, the authors consider two different particular examples where the oracle learner can effectively solve the prediction problem, while the standard learner fails to do so (Section 4).  I checked all the proofs appearing in the main part of the paper and they are correct.
While the main topic of the paper seems to be important and interesting, I was not impressed with the presented results. First of all, the assumption on the structure of $P$ is too strong for my taste: not only did we assume that such a factorization holds, but we also assumed that the oracle learner knows *exactly* the sufficient statistic $t$. I think this is a rather unrealistic situation. Second, the main theoretical result (Theorem 3) trivially follows the standard proofs of the lower bounds in the supervised learning (Chapter 14 in Devroye et al.). The only difference is captured in the extension of the fat-shattering dimension of Definition 1, but apart from that these are all well-known steps: let a marginal be concentrated on shattered points, then use probabilistic method, and play with the labels on the points not appearing in the training sample. In other words, I would say Theorem 3 does not contain any significant novel ideas. Finally, Section 4 looks confusing to me: the authors did not provide the original example of Niyogi in the "Manifold Learning" paragraph, which makes it hard to assess the novelty of the current paper in this context; "Group-Invariant features" paragraph is overly dense and is not supported with clear and intuitive discussions.   Concluding, I am not sure if this work provides a sufficient contribution to the literature (neither in terms of the techniques used, nor in terms of the results proved and conclusions made).   DETAILED COMMENTS  (0) Line 207 and 221-222. The upper bound of Devroy et al. holds for the *excess loss*, which is the difference between the loss of the learning algorithm and the smallest loss of the functions from the class. Meanwhile, the authors are using it to upper bound the loss of the learner itself. This is true only when we are in the separable realizable situation, meaning the hypotheses class contains the BEST classifier, which achieves zero error overall. This was not explicitly stated in the setting (at least, I did not notice it). (1) Words "fiber sub-bundle" in the abstract look mysterious... (2) Lines 42-43: Devroy et al. does not contain an overview of Rademacher complexities, only VC. (3) Line 64: not clear what do the authors call "a full joint model" (4) Section 2. I had a hard time guessing the meaning of $f_p$. In the end (from the proof of Theorem 3) I conclude that in all cases $f_p(x) = P(Y=1|X=x)$. This should be stated explicitly!  (5) Line 110. p(y|x) = E_p(y|x) should be replaced with p(y = 1|x) = E_p(y|x). (6) It is better to present or sketch the original definition of a fat-shattering dimension somewhere around Definition 1.  (7) Line 173. How can $P$ be an element of $P_t x C$ ?  (8) Equations (3) and (4): parenthesis missing, in (3) dot should be switched to comma. (9) line 212: "stronger notion of a stronger notion of..." (10) I did not get a purpose of a footnote (3) (11) lines 229-230. Better to state explicitly that this is due to probabilistic method, which is a standard tool used in these situations. Refer to the same book of Devroye, Chapter 14 if necessary. (12) Line 233. p(v_p(x)|x) should be replaced with p(y = v_p(x)|x) (13) Definition of $V_l$. Note that the r.h.s. depends on $i$ and $j$ while the l.h.s. does not. Should be fixed. (14) L.h.s. of the first inequality after line 236: the authors forgot to put $1/2^{2^m} \sum_{p\in P_{2m}}$.  (15) Line 246. $\bar{z}(\bar{x})$ should be replaced with $\bar{z}_p(\bar{x})$  =======UPDATE======= I have carefully read the rebuttal. unfortunately, I am going to keep my initial decisions.  The main reason for this is that I still think the assumptions considered in the paper are too strong. The authors are comparing the performance of a standard supervised algorithm to the learner which knows the structure of the problem, including the $t$ function. It is not surprising to me at all that in this case the "smart" learner will easily outperform the standard SL algorithm. In my point of view,  one could hope to *learn* the $t$ function during the training, but for sure it is unrealistic to assume that the algorithms *precisely knows* $t$ in advance. Thus I think the contribution of the paper is  marginal.


Average-case hardness of RIP certification
The restricted isometry property (RIP) for design matrices gives guarantees for optimal recovery in sparse linear models. It is of high interest in compressed sensing and statistical learning. This property is particularly important for computationally efficient recovery methods. As a consequence, even though it is in general NP-hard to check that RIP holds, there have been substantial efforts to find tractable proxies for it. These would allow the construction of RIP matrices and the polynomial-time verification of RIP given an arbitrary matrix. We consider the framework of average-case certifiers, that never wrongly declare that a matrix is RIP, while being often correct for random instances. While there are such functions which are tractable in a suboptimal parameter regime, we show that this is a computationally hard task in any better regime. Our results are based on a new, weaker assumption on the problem of detecting dense subgraphs.

A matrix of size (n,p), whose coefficients are independently drawn from suitable random distributions, satisfies with high probability the RIP property with parameters k,t, provided that n >> k log p / (t*t). The question considered in this article is: is there a polynomial-time algorithm that, given such a random matrix, is able to certify, in at least 2/3 of the cases, that this matrix indeed satisfies RIP? The answer is known to be positive when n is of the order of (k*k) log p / (t*t). This article shows that, for any alpha < 1, there is a sequence of (n,k,t) going to infinity, such that n >> k^(1+alpha) / (t*t), for which no certifier exists. The result holds under the assumption that a problem of dense subgraph testing can also not be solved with relatively large probability in polynomial time, in a certain regime.
I think that the result of this article is interesting and elegant. I would be happy to see it published. However, the proofs, in the appendix, contain a number of typos that need to be corrected. There is also an inequality that I do not understand, at the end of the proof of the main theorem; I would appreciate additional explanations about it.  Regarding the presentation, I find the article well-written. I have two regrets, but they are relatively minor: - I think it would be better to underline before the very end of the article that the proof does not currently work in the regime n >> k^(1+alpha) and theta constant. - I would have liked the authors to give a general picture of the proof in the main part of the article, more detailed that the paragraph that follows Theorem 4.  Typos and remarks: - Line 16: "ee" -> "see". - Theorem (informal): I think the statement would be more clear with "for any alpha" at the beginning of the sentence. - Line 80: "an a" -> "a". - "measureable" -> "measurable" (appears twice). - Line 178: "lim" -> "lim inf". - Line 256: I think A_G has not been defined. - Lines 284/285, "shows through a reduction to the planted dense subgraph problem": this part of the sentence is not clear. - In the appendix, equation after line 9: "\approx" -> "<<" ? - Algorithm 1: I think the definition of k should appear before the one of l. - Line 14: the coefficients of X to not exactly have distribution \tilde Q, because of the averaging. - Equation after line 14: it does not necessarily go to zero. I think we only know that the lim sup is at most 1/3. - Line 15: the statement holds only "with high probability". - Equation after line 16: I do not think that the second part is necessarily true. You have to specify how you choose theta. - Equation after line 18: the lim inf must actually be zero. - Equation after line 23 should be broken in two lines. - Equation after line 34 is only valid for epsilon under some constant value. - Equation (3): I do not understand where the k comes from in the denominator, inside the exponential. - Line 46: shouldn't v be padded with zeros, so as to be of size p? - Equation after line 49: I think that the zero-norm of \tilde v should be replaced with the cardinality of \tilde S. - Equation after line 49 (this is the one that I do not understand): from what I understand, the authors use as a lower bound for the 1-norm the sum of Z^(a,b) v^(b). Is my understanding correct? Because then, aren't there cross terms Z^(a,b) v^(b')? - Line 53: I think the authors could give a slightly more detailed justification, because \tilde v and \tilde X are not independent. - Line 57: I think a l^2 is missing in the denominator. - Statement of Lemma 1: I think the three inequalities are in the wrong direction. I also have the impression that the right hand sides should be squared (because of the application of Chebyshev's inequality). - Equation after line 67: in the first line, the 'v' should be a 'u'. At the end, there should be only one dot. - Equation after line 78: the inequality should be reversed. - Equation after line 90: I think that there should not be a square in the right hand side. I also do not think that the second and third terms of the middle part can be directly majorized by 1/2 |u_J^TAu_J|. - Line 97: "O(n^2p)" -> "O(np^2)"? - Line 98: "lim" -> "lim inf". - Line 101: "and fact that" -> "and the fact that". - Equation after line 102: shouldn't the term in the exponential be divided by 4, as in the equation after line 85, where the denominator is 256, and not 64? - Line 107: "R" -> "mathbb{R}". - Lines 114 to 116: is b equal to mu? - Equation after line 127: I think that the second equality is an inequality. - Statement of Lemma 6, and last equation of the proof: I think (1-n^(1-gamma)) is (1-n^(1-gamma))^(-1). - Line 146: "smaller than ||U||_0" -> "smaller than ||U||_\infty". - Line 149: "any one" -> "any of the".


Pairwise Choice Markov Chains
As datasets capturing human choices grow in richness and scale-particularly in online domains-there is an increasing need for choice models that escape traditional choice-theoretic axioms such as regularity, stochastic transitivity, and Luce's choice axiom. In this work we introduce the Pairwise Choice Markov Chain (PCMC) model of discrete choice, an inferentially tractable model that does not assume any of the above axioms while still satisfying the foundational axiom of uniform expansion, a considerably weaker assumption than Luce's choice axiom. We show that the PCMC model significantly outperforms both the Multinomial Logit (MNL) model and a mixed MNL (MMNL) model in prediction tasks on both synthetic and empirical datasets known to exhibit violations of Luce's axiom. Our analysis also synthesizes several recent observations connecting the Multinomial Logit model and Markov chains; the PCMC model retains the Multinomial Logit model as a special case.
stochastic transitivity does not imply the choice axiom; Thurstone's model exhibits strict stochastic transitivity.
Many choice theorists and empiricists, including Luce, have noted that the choice axiom and stochastic transitivity are strong assumptions that do not hold for empirical choice data [9,12,13,26,28]. A range of discrete choice models striving to escape the confines of the choice axiom have emerged over the years. The most popular of these models have been Elimination by Aspects [29], mixed MNL (MMNL) [6], and nested MNL [22]. Inference is practically difficult for all three of these models [15,23]. Additionally, Elimination by Aspects and the MMNL model also both exhibit the rigid property of regularity, defined below.
A broad, important class of models in the study of discrete choice is the class of random utility models (RUMs) [4,20]. A RUM affiliates with each i ∈ U a random variable X i and defines for each subset S ⊆ U the probability Pr(i chosen from S) = Pr(X i ≥ X j , ∀j ∈ S). An independent RUM has independent X i . RUMs assume neither choice axiom nor stochastic transitivity. Thurstone's Case V model and the BTL model are both independent RUMs; the Elimination by Aspects and MMNL models are both RUMs. A major result by McFadden and Train establishes that for any RUM there exists a MMNL model that can approximate the choice probabilities of that RUM to within an arbitrary error [23], a strong result about the generality of MMNL models. The nested MNL model, meanwhile, is not a RUM.
Although RUMs need not exhibit stochastic transitivity, they still exhibit the weaker property of regularity: for any choice sets A, B where A ⊆ B, p xA ≥ p xB . Regularity may at first seem intuitively pleasing, but it prevents models from expressing framing effects [12] and other empirical observations from modern behavior economics [28]. This rigidity motivates us to contribute a new model of discrete choice that escapes historically common assumptions while still furnishing enough structure to be inferentially tractable.
The present work. In this work we introduce a conceptually simple and inferentially tractable model of discrete choice that we call the PCMC model. The parameters of the PCMC model are the off-diagonal entries of a rate matrix Q indexed by U . The PCMC model affiliates each subset S of the alternatives with a continuous time Markov chain (CTMC) on S with transition rate matrix Q S , whose off-diagonal entries are entries of Q indexed by pairs of items in S. The model defines p iS , the selection probability of alternative i ∈ S, as the probability mass of alternative i ∈ S of the stationary distribution of the CTMC on S.
The transition rates of these CTMCs can be interpreted as measures of preferences between pairs of alternatives. Special cases of the model use pairwise choice probabilities as transition rates, and as a result the PCMC model extends arbitrary models of pairwise choice to models of setwise choice. Indeed, we show that when the matrix Q is parameterized with the pairwise selection probabilities of a BTL pairwise choice model, the PCMC model reduces to an MNL model. Recent parameterizations of non-transitive pairwise probabilities such as the Blade-Chest model [8] can be usefully employed to reduce the number of free parameters of the PCMC model. Our PCMC model can be thought of as building upon the observation underlying the recently introduced Iterative Luce Spectral Ranking (I-LSR) procedure for efficiently finding the maximum likelihood estimate for parameters of MNL models [21]. The analysis of I-LSR is precisely analyzing a PCMC model in the special case where the matrix Q has been parameterized by BTL. In that case the stationary distribution of the chain is found to satisfy the stationary conditions of the MNL likelihood function, establishing a strong connection between MNL models and Markov chains. The PCMC model generalizes that connection.
Other recent connections between the MNL model and Markov chains include the work on RankCentrality [24], which employs a discrete time Markov chain for inference in the place of I-LSR's continuous time chain, in the special case where all data are pairwise comparisons.
Separate recent work has contributed a different discrete time Markov chain model of "choice substitution" capable of approximating any RUM [3], a related problem but one with a strong focus on ordered preferences. Lastly, recent work by Kumar et al. explores conditions under which a probability distribution over discrete items can be expressed as the stationary distribution of a discrete time Markov chain with "score" functions similar to the "quality" parameters in an MNL model [17].
The PCMC model is not a RUM, and in general does not exhibit stochastic transitivity, regularity, or the choice axiom. We find that the PCMC model does, however, obey the lesser known but fundamental axiom of uniform expansion, a weakened version of Luce's choice axiom proposed by Yellott that implies the choice axiom for independent RUMs [30]. In this work we define a convenient structural property termed contractibility, for which uniform expansion is a special case, and we show that the PCMC model exhibits contractibility. Of the models mentioned above, only Elimination by Aspects exhibits uniform expansion without being an independent RUM. Elimination by Aspects obeys regularity, which the PCMC model does not; as such, the PCMC model is uniquely positioned in the literature of axiomatic discrete choice, minimally satisfying uniform expansion without the other aforementioned axioms.
After presenting the model and its properties, we investigate choice predictions from our model on two empirical choice datasets as well as diverse synthetic datasets. The empirical choice datasets concern transportation choices made on commuting and shopping trips in San Francisco. Inference on synthetic data shows that PCMC is competitive with MNL when Luce's choice axiom holds, while PCMC outperforms MNL when the axiom does not hold. More significantly, for both of the empirical datasets we find that a learned PCMC model predicts empirical choices significantly better than a learned MNL model. A Pairwise Choice Markov Chain (PCMC) model defines the selection probability p iS , the probability of choosing i from S ⊆ U , as the probability mass on alternative i ∈ S of the stationary distribution of a continuous time Markov chain (CTMC) on the set of alternatives S. The model's parameters are the off-diagonal entries q ij of rate matrix Q indexed by pairs of elements in U . See Figure 1 for a diagram. We impose the constraint q ij + q ji ≥ 1 for all pairs (i, j), which ensures irreducibility of the chain for all S.
Given a query set S ⊆ U , we construct Q S by restricting the rows and columns of Q to elements in S and setting q ii = − j∈S\i q ij for each i ∈ S. Let π S = {π S (i)} i∈S be the stationary distribution of the corresponding CTMC on S, and let π S (A) = x∈A π S (x). We define the choice probability p iS := π S (i), and now show that the PCMC model is well defined. Proposition 1. The choice probabilities p iS are well defined for all i ∈ S, all S ⊆ U of a finite U .
Proof. We need only to show that there is a single closed communicating class. Because S is finite, there must be at least one closed communicating class. Suppose the chain had more than one closed communicating class and that i ∈ S and j ∈ S were in different closed communicating classes. But q ij +q ji ≥ 1, so at least one of q ij and q ji is strictly positive and the chain can switch communicating classes through the transition with strictly positive rate, a contradiction. While the support of π S is the single closed communicating class, S may have transient states corresponding to alternatives with selection probability 0. Note that irreducibility argument needs only that q ij + q ji be positive, not necessarily at least 1 as imposed in the model definition. One could simply constrain q ij + q ji ≥ for some positive . However, multiplying all entriesof Q by some c > 0 does not affect the stationary distribution of the corresponding CTMC, so multiplication by 1/ gives a Q with the same selection probabilities.
In the subsections that follow, we develop key properties of the model. We begin by showing how assigning Q according a Bradley-Terry-Luce (BTL) pairwise model results in the PCMC model being equivalent to BTL's canonical extension, the Multinomial Logit (MNL) set-wise model. We then construct a Q for which the PCMC model is neither regular nor a RUM.

This paper considers the problem of developing flexible choice models that are not constrained to satisfy traditional, restrictive choice axioms (such as Luce’s axiom of independence of irrelevant attributes, IIA), but that can be tractably inferred from data.   A (discrete) choice model over n items specifies probabilities of the form p(i,S) = Prob( i chosen from S ) for each subset of items S \subseteq [n] and each item i \in S. One of the most widely used models of discrete choice is the multinomial logit (MNL) choice model, which can be inferred efficiently from data but which is constrained to satisfy IIA and other restrictive assumptions.    The paper proposes a new Markov chain based model of discrete choice that is parametrized by a (n x n) pairwise selection probability matrix. The model avoids several of the earlier restrictive assumptions, but is shown to satisfy an interesting property termed contractibility, which in turn also implies a reasonable property of uniform expansion. Parameter estimation in the model is done by maximum likelihood (the log-likelihood function is non-concave in general, but the experiments suggest that good parameters are learned). Previous models that avoid the above restrictive assumptions either do not satisfy even the uniform expansion property or are computationally difficult to learn, and so the proposed model offers unique properties in comparison to previous models. The proposed model also generalizes a recent Markov chain based formulation of the MNL model, which is subsumed as a special case.  The proposed model is applied to two real data sets on transportation choices in the Bay Area, and in both cases, shows better predictive ability than the classical MNL model.
This is a strong paper, and additionally, is beautifully written and presented. It is one of the rare papers where I have very few comments or suggestions to make (just one question below). Truly beautiful work, and a pleasure to read!  Ques: What is the running time of the proposed parameter estimation algorithm on the two data sets, and how does this compare with the running time for learning the MNL model? It would be nice to give details on this in the author rebuttal if possible, and to include them in the final version if accepted. In future work or a longer version, it would also be nice to include such comparisons with other models such as nested MNL etc.  ---- After author response -----  I have read the author response and it addresses my question.


Yggdrasil: An Optimized System for Training Deep Decision Trees at Scale
Deep distributed decision trees and tree ensembles have grown in importance due to the need to model increasingly large datasets. However, PLANET, the standard distributed tree learning algorithm implemented in systems such as XGBOOST and Spark MLLIB, scales poorly as data dimensionality and tree depths grow. We present YGGDRASIL, a new distributed tree learning method that outperforms existing methods by up to 24×. Unlike PLANET, YGGDRASIL is based on vertical partitioning of the data (i.e., partitioning by feature), along with a set of optimized data structures to reduce the CPU and communication costs of training. YGGDRASIL (1) trains directly on compressed data for compressible features and labels; (2) introduces efficient data structures for training on uncompressed data; and (3) minimizes communication between nodes by using sparse bitvectors. Moreover, while PLANET approximates split points through feature binning, YG-GDRASIL does not require binning, and we analytically characterize the impact of this approximation. We evaluate YGGDRASIL against the MNIST 8M dataset and a high-dimensional dataset at Yahoo; for both, YGGDRASIL is faster by up to an order of magnitude.

This paper presents a new system for performing distributed computing in an efficient way for decision trees. This is done in a classical master-workers setting. Compared to Planet, the algorithm used in standard implementation, the proposed method splits data vertically, distributing distinct features to workers (as opposed to distributing learning instances). The purpose of the method is to solve the bottleneck created by the expensive communication necessary for horizontal approaches (like Planet), scaling up to a larger number of features and bigger decision trees.
The paper is well written, and its structure is adapted to the content. Upon reading the paper, one might think that the contribution resides in the vertical splitting of the data over the workers, but the state of the art study presented later on shows that this idea by itself is not new. The novelty comes from associating it with data also distributed vertically, sparse bit vectors for inter-node communications, feature compression with custom data structures and training on compressed data. The paper shows formally and experimentally how the proposed heuristics significantly improve the communication between the nodes and speed up training.  The remark that using run-length encoding for the features allows them to hold in the L3 cache, thus decreasing the number of DRAM accesses, doesn't seem to always be true. The paper should explain in which conditions this is true (size of the cache, size of the data, number and type of features, etc.).  The heuristics are presented for a fixed type of architecture: one master connected to a certain number of workers in a star configuration. Is this the only setting used for large scale distributed computing, in machine learning or fields with similar needs? Would changing this architecture remove some of the existing bottlenecks?  The experimental results compare Yggdrasil against different implementations of Planet. On the two datasets the method shows important improvements in the computation time compared to other methods. Here, each of the datasets is used for a different task (classification and regression), so it is hard to tell if the speed gain is general or specific to the dataset and the task. It would be useful to see the performance of Yggdrasil on some other datasets as well.  Yggdrasil could be of impact in the community making use of distributed architectures. While the experimental results show a clear improvement over state of the art methods, they would be more convincing over a larger number of datasets.  Minor remarks: I am not sure that the NIPS format allows to surround figures (Figure 3 in this case) with text. End of Eq. (1): x should maybe be in bold?


Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
We study optimization of finite sums of geodesically smooth functions on Riemannian manifolds. Although variance reduction techniques for optimizing finite-sums have witnessed tremendous attention in the recent years, existing work is limited to vector space problems. We introduce Riemannian SVRG (RSVRG), a new variance reduced Riemannian optimization method. We analyze RSVRG for both geodesically convex and nonconvex (smooth) functions. Our analysis reveals that RSVRG inherits advantages of the usual SVRG method, but with factors depending on curvature of the manifold that influence its convergence. To our knowledge, RSVRG is the first provably fast stochastic Riemannian method. Moreover, our paper presents the first non-asymptotic complexity analysis (novel even for the batch setting) for nonconvex Riemannian optimization. Our results have several implications; for instance, they offer a Riemannian perspective on variance reduced PCA, which promises a short, transparent convergence analysis.

This paper develops RSVRG, a variance-reduced stochastic gradient method for minimizing finite sums of functions on Riemannian manifolds. The paper establishes that the approach can converge linearly, even for geodesically nonconvex functions, and that the complexity of the method depends on the geometry of the manifold. The convergence results are nonasymptotic, and the complexity results use the IFO paradigm. The paper provides experiments for two applications. One application is PCA and the other is computing the Riemannian centroid of symmetric positive definite matrices. Through these experiments, the paper illustrates the benefit of variance reduction.
This paper addresses a topic that was proposed as future work in [32], and by design, much of its contents are adaptations of results found in [14] and [21]. Thus, the analysis is incremental. Nonetheless, the paper contributes a manifoldized algorithm and bridges a gap between convex optimization and Riemannian optimization. Furthermore, the paper provides a useful lemma to analyze Riemannian methods, which can have a lasting impact.  My main concern with this paper is its presentation. The paper is elusive on details, which undermines its technical quality, omits an important reference, and suffers from language issues. In Theorem 4, the constants mu1 and nu1 should be defined (are they the same as the constants in [21]?). For the PCA application, the matrix Z should be related to the problem (are the vectors z the columns of Z as in [25]?). For both applications, it should be mentioned how the accuracy is measured (accuracy is the ordinate of most of the plots). For the centroid application, the parameters N and Q should be defined (is N=n?). The paper should expound on the exponential and parallel transport mappings (the most important parts of RSVRG) for each of the applications: granted, for the PCA application, the paper cites an entire textbook for the mappings, but a citation like this is more pleasant with page numbers (I could not find the relevant results). The reference for the IFO framework should be included; the reference is "A Lower Bound for the Optimization of Finite Sums" by Alekh Agarwal and Léon Bottou. Finally, the paper would benefit from a good proof read. Here are some of the language issues:  Line 48: "developing fast stochastic for it". Maybe this should read "developing fast stochastic methods for it". Line 56-57: "we generalize convergence analysis". This should read "we generalize the convergence analysis"; there are many missing "the's" in the paper. Line 59: "method this is provably". Perhaps "this" should be replaced by "that", or the sentence should be reworded. Line 130: "takes and i". Not "and" but "an". Line 155: "an fi minimize all". Not "minimize" but "minimizes". Theorem 2: The first sentence is incomplete. Maybe "If" should be replaced by "Suppose" (as in the restatement of the theorem in the supplementary material).  Here are some other issues: In the experiments for PCA, the methodology for estimating the number of accuracy-doubling epochs should be described. This number becomes less for RSVRG than for VR-PCA as the eigengap decreases, and it should be mentioned whether the number of IFO calls follows this tendency for a given accuracy (the comparison is provided only for a single, relatively large delta). It should be explained why the IFO complexity provides a fair comparison (RSVRG requires evaluating not only gradients, but also exponential and parallel transport mappings - are the extra computations worth it?).  More issues:  Equation 8: The index i on each z is missing.  Line 435 (in the supplementary material): "Corollary ??".


Examples are not Enough, Learn to Criticize! Criticism for Interpretability
Example-based explanations are widely used in the effort to improve the interpretability of highly complex distributions. However, prototypes alone are rarely sufficient to represent the gist of the complexity. In order for users to construct better mental models and understand complex data distributions, we also need criticism to explain what are not captured by prototypes. Motivated by the Bayesian model criticism framework, we develop MMD-critic which efficiently learns prototypes and criticism, designed to aid human interpretability. A human subject pilot study shows that the MMD-critic selects prototypes and criticism that are useful to facilitate human understanding and reasoning. We also evaluate the prototypes selected by MMD-critic via a nearest prototype classifier, showing competitive performance compared to baselines.

The authors give a presentation of the mmd-critic algorithm, which selects prototypes and criticism for human interpretable machine learning by optimizing a statistic called the maximum mean discrepancy. This is a novel application of a method that was previously applied to model comparison and comparing models to input data. They describe the model and show a proof for submodularity that ensures the method is scalable. They tested the method against other prototype-based machine learning baseline methods from Bien and Tibshirani (2011) for different numbers of prototypes on the USPS handwritten digit database. The local version of MMD-critic showed superior performance to the baselines, mainly for less than 1000 prototypes. A qualitative test was done on images of dogs. Finally, they ran a pilot study to collect measures of interpretability for MMD-critic. They concluded that people performed best when given both prototypes and criticisms.
The authors explore the compelling question of how to develop interpretable machine learning methods using prototypes and criticisms. The paper was well written and clear, even for a non-expert in the field like myself. The mathematical results appear to be sound.  It is hard for me to assess the originality of the work in the field of machine learning, but I imagine that there is work on training with both positive and negative examples. At the very least, within the human category learning literature the issue of learning a concept through examples of the concept and non-examples has been explored. Given that the authors are interested in human interpretable machine learning methods, they should investigate this work in psychology.  Relatedly, the methods they compare mmd-criticism to do not have criticisms (negative examples), which makes direct comparison between the algorithms difficult to assess. This is also the case for the human experiments: participants may have done well if they were simply given randomly chosen criticisms and not those chosen by mmd-criticism. In short, the authors have certainly shown that computer and human performance is better given prototypes and criticisms rather than only prototypes. It would have been interesting to see how mmd-critic performs against other algorithms that also provide both types of data.  Smaller points: - Eq 4 seems to be missing "k(" - In figure 1, does the value on the x axis include both prototypes and criticisms for the mmd-critic model?


Scalable Adaptive Stochastic Optimization Using Random Projections
Adaptive stochastic gradient methods such as ADAGRAD have gained popularity in particular for training deep neural networks. The most commonly used and studied variant maintains a diagonal matrix approximation to second order information by accumulating past gradients which are used to tune the step size adaptively. In certain situations the full-matrix variant of ADAGRAD is expected to attain better performance, however in high dimensions it is computationally impractical. We present ADA-LR and RADAGRAD two computationally efficient approximations to full-matrix ADAGRAD based on randomized dimensionality reduction. They are able to capture dependencies between features and achieve similar performance to full-matrix ADAGRAD but at a much smaller computational cost. We show that the regret of ADA-LR is close to the regret of full-matrix ADAGRAD which can have an up-to exponentially smaller dependence on the dimension than the diagonal variant. Empirically, we show that ADA-LR and RADAGRAD perform similarly to full-matrix ADAGRAD. On the task of training convolutional neural networks as well as recurrent neural networks, RADAGRAD achieves faster convergence than diagonal ADAGRAD.

This paper formally proposes two algorithms: ADA-LR and RADAGRAD, which are computationally efficient approximations of ADAGRAD when using a full matrix approximation for the proximal term. The authors use ADA-FULL for this full-matrix method and use ADAGRAD to refer to the most common variant where one uses a diagonal matrix approximation. Though ADAGRAD performs well in practice, the authors argue that when data has low effective rank, the regret of ADAGRAD is substantially worse than ADA-FULL, and the problem is exacerbated in high dimensions (e.g., the combined weight vector for deep neural networks). This therefore motivates the use of their two algorithms to approximate ADA-FULL. ADA-LR is an approximation based on random projections, and RADAGRAD takes this a step further by using more randomized methods. The authors also identify additional improvements to RADAGRAD for practical performance benefits. Their experimental results on image recognition (using convolutional neural networks) and language modeling (using long short term memory networks) show that the performance of their algorithms is comparable to or superior than the state of the art.
       I am generally impressed with this paper. The introduction gives convincing arguments for the need to modify ADAGRAD/ADA-FULL in the prevalence of low effective rank data. Then the paper introduces and describes the new algorithms as a series of matrix operations to approximate ADA-FULL. Then it provides some theoretical results (deferring proofs to the appendix) and presents three sets of experiments. This is a logical progression of the material.              In the related work section, you might wish to remove the second paragraph in favor of other references that argue about the prevalence of low effective rank data. The second paragraph, while certainly conceptually related to the paper, doesn't have anything saying its explicit relation. For instance, having something like "variance reduced SGD algorithms have vastly superior performance BUT perform extremely poorly on low effective rank data" would be helpful to understand its context. I suggest this because the paper's contribution is only going to be as good as the prevalence of low effective rank data in the real world, and this is the recurring theme that must be emphasized.              The algorithms appear useful and insightful, and involve "just" a series of matrix operations simple enough so that an alert reader should be able to code it up. It seems a little awkward to include ADA-VR when, as far as I can tell, there's virtually no reason to use it over RADAGRAD, but it makes the paper's story proceed nicely so I guess it's OK. The table in Appendix A is enormously helpful. Thank you for including this.              The experiments the authors chose to run seem reasonable to me given the research goals (and the 9-page space constraint). The first experiment is logical to include since it provides the best-case scenario for the algortihms, and they certainly perform well. The second experiment presented in Figure 2 is, to me, the most interesting one of the paper. My main concern there pertains to your comment about using RADAGRAD only for the convolutional layers. It seems like a fairer comparision would be to use RADAGRAD for everything or ADAGRAD for everything? I think more discussion about this is worthier than the earlier discussion about the implementation; there's little reason to talk about the code beyond that it was implemented in Lasagne/Theano/Python.              I am also intrigued by the third experiment due to the growing importance of LSTMs nowadays. However, the performance benefits over ADAGRAD+SVRG seem extremely minor, even when taking into account the log scale. Is there a way to explain intuitively what this extra benefit means? You might explain this and delete the second paragraph in Section 5, which doesn't seem to make sense for the paper (the forgetting schemes were never discussed earlier).              Despite some nitpicks over the experiments, as mentioned earlier I have a positive impression on this paper, and lean towards its acceptance to NIPS. It introduces an important extension to the widely-used ADAGRAD function and argues that these algorithms are needed when we have low effective rank data. While it is not as groundbreaking as the original ADAGRAD paper, I believe the research contribution is substantial enough to warrant consideration to NIPS.              Minor comments:       In line 108, you might wish to say "borrowing the $g_{1:T,j}$ notation" or something like that, because the notation is never explicitly defined in this paper, but is used later (e.g., in Equation 2).       In lines 124 through 128 and including Equation 2, I am unable to derive the first part of Equation 2. I am also unsure why you assume $g_t \propto Mx_t$, because M is later revealed to be a scalar, so why the need for the proportional symbol? That constant can be absorbed into M. I once thought it should be something like $\|g_t\|_2 \le M\|x_t\|_2$ but perhaps the authors could clarify.       In line 145, did you mean to say "applicable in this low-dimensional *space*"?       In line 192, "it's" should be "its".       In Figure 1, please be consistent and use ADA-FULL instead of ADA-F, since the former is used throughout the paper's text.       In Figure 1(b), you might consider using the same number of principal components for RADAGRAD and ADA-LR. I think one has 21 and another has 28, and I believe these values come from the top $\tau$ singular values, so you might want to make them equal, or otherwise explain your choice for $\tau$. I'm not sure why there's a difference right now. I'm assuming $\tau$ is just chosen heuristically?       In line 268, "A the end" should be "At the end".       In line 397 (in Appendix), the title of C.1 is misleading, it should be the regret bound for ADA-LR, not RADAGRAD.


Active Learning from Imperfect Labelers
We study active learning where the labeler can not only return incorrect labels but also abstain from labeling. We consider different noise and abstention conditions of the labeler. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under fairly natural assumptions on the noise and abstention rate of the labeler. This algorithm is adaptive in a sense that it can automatically request less queries with a more informed or less noisy labeler. We couple our algorithm with lower bounds to show that under some technical conditions, it achieves nearly optimal query complexity.

The paper studies active learning where the labelers may make errors and abstain from providing a label. The paper shows lower bounds and upper bounds on the number of queries needed in the one dimensional case and d dimensional case.  One possible motivation for this work is the results of Shah and Zhou “Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing” that shows that the only incentive compatible mechanism (under some assumptions) for crowed sourcing requires that workers abstain when they are uncertain.
 • You are making the assumption that the hypothesis is the epigraph of holder smooth function. How realistic is this assumption? For example, can the decision boundaries of neural network, decision trees and kernel models have these properties? • Line 120: you are making here the assumption that the distribution is uniform over the unit cube. This should be made explicitly. How sensitive are the results to this assumptions? For example, Freund, Seung, Shamir and Tishby in their work on the query by committee algorithm used an assumption that the distribution is within a constant from uniform. • Assumption 3 seems very restrictive since it assumes equality.  • Line 162 – can you show how does the assumption result in the decision boundary being a single point? • The technique uses membership queries and furthermore assume that the labeler is not consistent in the sense that when queried multiple times on the same point it may return different answers. Both assumptions are questionable. • Theorem 2: since the algorithm performs the significance test at every round, don’t you need to use the union bound to make sure that the significance test is true for all queries? In that case, the significance bound for the test should be updated to reflect the number of times it is being used. • Lines 232-234: you claim that errors are more harmful than missing labels. Is it a universal truth or specific to the current set of assumptions? • Theorem 9: the number of queries made is exponential in the dimension whereas standard PAC models are typically linear in the dimension. In what cases is it better to use your model?   In their response, the authors address many of my concerns. The constraints are not "as bad" as I thought they are. Nevertheless, there are still many constraints in this setting and no demonstration of a case in which these constraints hold. I think this paper can be much improved and would create bigger impact after such improvements are implemented.


Causal meets Submodular: Subset Selection with Directed Information
We study causal subset selection with Directed Information as the measure of prediction causality. Two typical tasks, causal sensor placement and covariate selection, are correspondingly formulated into cardinality constrained directed information maximizations. To attack the NP-hard problems, we show that the first problem is submodular while not necessarily monotonic. And the second one is "nearly" submodular. To substantiate the idea of approximate submodularity, we introduce a novel quantity, namely submodularity index (SmI), for general set functions. Moreover, we show that based on SmI, greedy algorithm has performance guarantee for the maximization of possibly non-monotonic and non-submodular functions, justifying its usage for a much broader class of problems. We evaluate the theoretical results with several case studies, and also illustrate the application of the subset selection to causal structure learning.

The authors study two related problems which they refer to as source detection (OPT1) and causal covariate selection (OPT2).  OPT1 is related to a body of work on sensor selection, but whereas past work typically use mutual information this work uses directed information.  Likewise problems like OPT2 have been studied for decades, though often with strong modeling assumptions.  The authors suggest using directed information for that too (it is a non-parametric method).  The authors show that OPT1 is submodular but not monotone (thus greedy search has performance guarantees), while OPT2 is monotone but generally not submodular.    The authors introduce a novel measure of how "submodular" a functions is (which they refer to as submodularity index SmI) and obtain performance bounds for maximization in OPT1 and OPT2 with cardinality constratints, where the bounds are a function of SmI.   They also analyze both simulated and real world data, showing performance of the algorithm and computing some empirically observed SmI values.
 Overall, I find the work promising and it contains a number of results which will appeal to the NIPS community.  However, I think a significant revision for OPT2 is needed for it to appear at NIPS.  They are also missing citations for work closely related to their work on OPT2.  There are multiple minor issues that should also be addressed, listed further below.  Major comments (What I like):  -- Both problems are interesting and relevant to NIPS. Especially OPT2, broadly speaking, has been of great concern and generated decades of research in various domains.  To approach it with a non-parametric method is quite appealing because then one needs not worry as much about strong modeling assumptions.     -- SmI is an intriguing way of assessing submodularity, though it makes sense.  It is highly context dependent (function of A and k), but that seems necessary given your later work and data analysis.  I like how the results are not limited to specific functions but could potentially be applied to a broad range of combinatorial problems.    -- The performance bounds are good.    -- It was applied to both simulated and real world data.  Aside from empirical values for SmI, it does give credence at least to employing greedy methods (especially when exact is computationally prohibitive).    Major comments (What I think needs to be modified to improve the work):  (A) My major concern is with regards to the usefulness of the bounds for OPT2.  While results for OPT1 are clearly valid (since is submodular), the usefulness of the bounds for OPT2 is unknown.  They are useful only when OPT2 exhibits "quasi-submodularity," but when does that happen?  Statements such as       (line 44) "is “nearly” submodular in particular cases"   (line 216) "For quasi-submodular functions having negative SmI, although a degraded guarantee is produced, the bound is only slightly deteriorated when SmI is close to zero."   (line 227) "Similarly, for quasi-submodular functions, the guarantee is degraded but not too much if SmD is close to 0. Note that the objective function of OPT2 fits into this category." are made throughout the paper but not justified or made precise.    "quasi-submodular functions" is defined as (line 163) "We call a function quasi-submodular if it has a negative but close to zero SmI."  The authors never give a precise definition of what consistutes "close."  They also never give examples of when OPT2 is in fact quasi-submodular except in the extreme case of Proposition 2 (line 131) where you have independence *conditioned on Y_t*.  [Please give an example that satisfies Prop 2.  Even when causal covariates are marginally independent (which is already restrictive), say Y_{t+1} is a linear combination of marginally independent X_{it}'s,  then conditional independence (prop 2) won't hold.]    This is not to say that the formulas obtained are not useful for monotonic non-submodular functions (especially since they are applicable to potentially a wide range of combinatorial optimization problems), but some non-trivially "quasi-submodular" optimization problems need to be identified, and I don't think OPT2 is a good candidate.  Perhaps for some certain classes of distributions it could be shown to be (and if the authors can do this, I would be happy with them keeping OPT2 in), but in general not.  The graphical model literature on structure learning has examples where there is pairwise independence (similar to I(X-->Y)=0, I(Z-->Y)=0)) but the joint relationship (so I(X,Z-->Y) can be made arbitrarily large when the alphabet is not restricted (eg using the XOR function and converting vector valued binary variables (with Y(i) = X(i) XOR Z(i), then convert the vector to a larger-alphabet variable)).       Also, note that while I am happy to see they applied their work to data, and that the randomized greedy did well, the fact that randomized greedy did well does not imply quasi-submodularity holds.  [Greedy algorithms (though not randomized greedy) have been widely used in graphical model structure learning literature, for random variables and processes.]  The experiments do provide evidence for using randomized greedy, which is good.  I was furthermore happy that they included some empirical values of SmI index.  And for row 2 of Table 1, since all the values are larger than -0.6321 the bound in corollary 1 (line 224-225) would be nontrivial (approximation ratio bigger than 0).  This gives a little evidence of quasi-submodularity, and it is interesting that k=2 is the worst, whereas others have positive or "close" to 0 values.  In addressing this major concern, if proving when quasi-submodularity holds (maybe you can define "close" to be relative to E[f(S^g)], such that close means bound in 224-225 is nontrivial) is too difficult, empirical evidence is fine, though it would need to be more extensive than for a single stock in one dataset.   (B) The paper is missing a reference that appears to be closely related, especially for OPT2.  Searching "Submodularity directed information" on arxiv yielded two works     * [highly relevant] "Bounded Degree Approximations of Stochastic Networks" http://arxiv.org/abs/1506.04767 by authors Quinn et al (you cite some of their other works) which appears related for the OPT2 problem.  Especially Section 5 of the arxiv version, as they do cardinality constrained causal covariate selection (OPT2; it looks like they are doing "approximation", though the objective functions look similar) and propose a approximate submodularity measure and obtain bounds on the deterministic greedy search.  Their approximate submodularity measure is not the same as in this submission (SmI) and I can't tell whether the one in Quinn et al would be useful for studying OPT1.  Also, wrt my comment above about "quasi-submodularity" being unjustified, this work by Quinn et al also does not theoretically justify when their approximate submodularity is useful (when their alpha is close to 1).      They mention the arxiv version (I don't see it published yet) is based on a conference paper "Optimal bounded-degree approximations of joint distributions of networks of stochastic processes" ISIT 2013, which looks to have the same results (regarding approximate submodularity), though the causal covariate selection looks a little different.      * [less relevant] a very recent paper (just a few months out before the NIPS 2016 initial deadline) "Submodular Variational Inference for Network Reconstruction" http://arxiv.org/abs/1603.08616 that reconstructs the network, but does so assuming a diffusion model.           Minor comments   -- mentioned in Major comment (A), please provide examples when Prop 2 holds, esp if you cannot more generally identify when OPT2 is quasi-submodular.    -- Question: This work employs the CTW algorithm.  But the available code http://web.stanford.edu/~tsachy/DIcode/download.htm and the original paper [9] are for pairs of processes.  In the submission, you are dealing with sets.  Did you modify the code to handle DI for sets?  If so, are you sure the theoretical guarantees extend?  Or did you do a mapping, such as converting three binary processes into one process of alphabet 8? This might be straightforward but I wonder if there would be performance hit (for finite length data). I'm assuming you did not just find pairwise relations for the original processes and then combine individual parents, as that could lead to significant errors.      -- Can you provide any guidance to the reader on selecting $k$?  I think this is important given that not only there are computation tradeoffs but the approximation coefficient is sensitive to $k$ (especially for OPT2)    -- Line 185 with Algorithm 1.  You should provide a sentance or two explaining how this random greedy relates to deterministic.  It is not obvious at first glance of the notation, even though the idea is simple.  "At each step, instead of picking the best, we pick one of the top k best" or something like that.      -- line 131 "any" can have multiple meanings.  I assume you want it to mean "every pair" though it can also mean "at least one pair".  I'd suggest using "every" to avoid confusion.    -- Provide references for quantities in Section 2.  At first I thought you were introducing them but they are in some works you cite.    -- Double check your Reference section; right now it is a little sloppy (eg line 313 "gaussian" should be capitalized, journals in 340 and 346 should be capitalized, conference title in 358 should be capitalized, you use anacronyms for some conference proceedings but then write out names for others)    -- Lemma 2 line 197 please clarify the notation.    Are these lower case y's processes?  If so why lower case?  And is Y={y_1, .. y_M} or a strict subset or a strict superset?    -- in line 201 you make a comment about being able to lower bound SmI for order 2, yet wasn't that the worst case you found empirically? (thus suggesting it would be challenging to do so)    -- fix grammer line 205 "with each element appears in"    -- formula below line 201, what is "p_1"    -- line 238, in formula for \gamma, if S^{g_k} means |S^{g_k}|=k, then just use k in the formula, and you might consider putting a table or graph to show behavior    -- line 249 the data you use is old -- if it appeared in another paper you should cite the source.      -- Section 5  for the data, I'm sure the stock data is not binary, perhaps you designed the Bayesian network data to be so; but describe how your discretize the data (especially since to just binary, not even quantizing)    -- line 257 you should not say "is sufficient" for real world data unless you have ground truth knowledge you are basing that off of.  Even convergence is not sufficient (maybe for length 5 it converges to one value, for length 6 another)     -- line 261 include an equation reference; "the 1/e bound" is vague


Full-Capacity Unitary Recurrent Neural Networks
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

This paper describes a method for training recurrent neural networks with unitary transition matrices that doesn't rely on restricting the set of unitary matrices that can be learned. The authors also provide some theoretical motivation for their method by proposing the use of Givens operators (similar to Givens rotations) to quantify the representational power of a certain type of unitary matrix. They use this mathematical machinery to show that the original parameterization of uRNNs is not able to represent the entire unitary group for N x N matrices for which N > 22. The authors propose to use a gradient descent optimization method that operates directly on the Stiefel manifold of unitary matrices and show on a couple of tasks that this method leads to results that are at least as good and sometimes clearly superior to results obtained with the original uRNN parameterization and LSTMs.
I think this is a strong paper in that it presents multiple theoretical and empirical contributions. The theoretical ideas and proposed optimization algorithm are in my eyes more impressive than the empirical work, which is also decent but could benefit from a more thorough analysis. In any case, I think it's a very nice continuation of the ideas presented in the original paper about uRNNs.  Except for some minor issues, the paper is well written in the sense that it was easy enough for me to follow the materials about Givens operators and how they can be used to quantify the representational capacity of a unitary matrix even though this specific subject matter was rather new to me.   The proofs in the supplementary material seems sound to me and are relatively simple.   The experimental section is a bit less convincing since all the tasks are either synthetic, or using real data in a setup that is still artificial. I don't have too many issues with the actual results, but the paper sometimes confused me a bit about the precise experimental designs and the interpretation of the results.  In line 206, the authors say that they investigated values of the matrix dimensionality N that are below, *at* and above the critical dimension of 22. The value 22 itself however, is not part of the set of dimensions investigated. I also would have expected a sudden increase in the performance gap between the restricted-capacity uRNN and the full-capacity uRNN after increasing N past the value 22. The difference in performance seems to greater for the lower dimension problems. I understand that it's hard to compare these results directly, because they're MSE scores for problems of different dimensionality, but a discussion about the lack of a sudden breakdown of the restricted model for N > 22 would in my opinion improve the paper.   In lines 211-212, the authors say they report the best test results over 100 epochs for several random initializations. It's not very nice to do this early stopping on the test data directly, although I suspect the results wouldn't be very difference. Knowing how many initializations where actually used and seeing standard errors or deviations in Table 2 would make this section of the paper a lot stronger.  The issue I have with the 'copy memory' task is that it's a negative results for the restricted uRNN. Looking at the original paper about uRNNs, I now know that the restricted variant can solve the task for T=500 when trained by the original authors but fails at T=1000 when trained by the authors of this paper. I'm not saying they didn't try to get the best results possible with the restricted version, but it would have been better if there was some indication of a thorough hyperparameter search for this baseline. That said, the results for the new method on this harder version of the task are still impressive and indicative of it's potential practical usefulness.  The speech task still feels a bit artificial to me as next frame prediction for speech features is not a very common application except perhaps for training generative models of speech. I'm not certain that this task requires an RNN to learn very long term dependencies ans neighboring frames from spectrograms with relatively small window sizes tend to be highly correlated. I find the results on this task somewhat surprising. The restricted uRNN seems to do surprisingly well in terms of MSE for up to 70 hidden units, while the full-capacity models always perform better on the other metrics. I wouldn't claim that the full-capacity model clearly outperforms the restricted variant here. It would also have been interesting to see the performance of much larger RNNs of 500 hidden units. Again, this task still provides evidence that the optimization method works and clearly outperforms the LSTMs for models of this size.   While it may seem I have a lot of issues with the quality of the individual experiments, I think that taken together they still provide a decent investigation of the methods presented. However, I also think that the paper should improve the discussion of the results and could benefit from some additional details that would make replication of the results easier.


On Multiplicative Integration with Recurrent Neural Networks
We introduce a general and simple structural design called "Multiplicative Integration" (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.

In this paper the authors advocate for the use of multiplicative interactions in RNNs. The sum between the projected hidden state and the input embedding is replaced by a component wise product. They show results on a large variety of tasks with small but consistent improvements. They also claim faster convergence.
My biggest concern about this work is the lack of novelty. Despite the claimed differences, the proposed method is a special case of what proposed in [10]. I doubt that the slight different parameterization (remove one factor-hidden matrix and introduce more bias terms) makes much difference. I strongly suspect that the improved performance is due to better optimization (HF has proven to be very brittle).  I also found weak the argument for which gating makes gradients flow better because there is no guarantee this is going to happen. The gradient has an additional term it's multiplied by, There is no guarantee that the Jacobian between two consecutive hidden states has unitary eigenvalues...  The experimental section is solid, although improvements are rather marginal yet consistent.  -------------------------------------------------- I have read the rebuttal. I sincerely apologize with the authors. There is no fatal flaw in their paper, I just clicked that by mistake. Apologies.  I still think that the paper lacks novelty, however.  [10] proposes multiplicative interactions for a plain RNN but there is no reason why the same would not apply to other types of RNNs like LSTMs. Multiplicative interactions apply to anything really (true for [10] and this paper). In [10] they had: h_t = non_linearity( V ((W_x x) .* (W_h h_{t-1})) ) here the authors propose: h_t = non_linearity( \alpha .* ((W_x x) .* (W_h h_{t-1})) + bias_terms) I do understand that this is a simplification of [10], but is it so different? The difference is removing matrix V above and introducing bias terms. If they authors acknowledged more and analyzed the effect of each of these modifications, it would have been more informative in my opinion.  That said, this paper has pretty solid empirical validation which compensates the lack of novelty. Based on this, I'd certainly recommend acceptance, but I would be quite hesitant about promoting this to an oral.


Distributed Flexible Nonlinear Tensor Factorization
Tensor factorization is a powerful tool to analyse multi-way data. Recently proposed nonlinear factorization methods, although capable of capturing complex relationships, are computationally quite expensive and may suffer a severe learning bias in case of extreme data sparsity. Therefore, we propose a distributed, flexible nonlinear tensor factorization model, which avoids the expensive computations and structural restrictions of the Kronecker-product in the existing TGP formulations, allowing an arbitrary subset of tensorial entries to be selected for training. Meanwhile, we derive a tractable and tight variational evidence lower bound (ELBO) that enables highly decoupled, parallel computations and high-quality inference. Based on the new bound, we develop a distributed, key-value-free inference algorithm in the MAPREDUCE framework, which can fully exploit the memory cache mechanism in fast MAPREDUCE systems such as SPARK. Experiments demonstrate the advantages of our method over several state-of-the-art approaches, in terms of both predictive performance and computational efficiency.

This paper proposes a tensor factorization algorithm that can be viewed as a tensor completion problem, in which a ``low-rank'' decomposition of the tensor is constructed using partial observations of the tensor.   It is based on the Tucker decomposition with appropriate priors for the Tucker factors $U$.  It considers both binary and continuous tensors. For continuous tensors the formulation is based on the model of equation 2, where the kernelized prior is based defined  in lines 113--120. Furthermore because the covariance matrix $K$ that appears in EQ 2 is very large, an approximation by using the scheme in 18.   A fixed-step gradient descent scheme is used (combined with L-BFGS). Additional optimizations are discussed for use on a distributed architecture using the MapReduce and the SPARK system.   The paper concludes with numerical experiments that demonstrate the performance of the algorithm in terms of accuracy of the approximation and AUC values for several datasets.
REVIEW  The results in Figure 2 look quite good, but why? Is it because kernelization or is it because of the simpler covariance structure? Overall no explanation on why the method performs better is given.    Timing comparisons with the other factorizations are not presented, so is the method faster? If not, what happens if we increase the ranks for the other methods. If there are so many fudging factors, where the other methods similarly optimized?  Not sure that the bounds are useful. How are they used in the algorithm?  There are numerous parameters: prior for W core tensor, kernel form and kernel parameters, $\beta$ for observations, the number of inducing points, regularization for $U$ (first term in Eq 2), the size $p$ for the B matrix. This seems an almost impossible method to use, how do you choose all these parameters? This formulation introduces more non-convexity with having to solve for the inducing points B.   Also, how is the algorithm terminated? In the experimental session the splitting between testing and non-testing is arbitrary.    SECONDARY COMMENTS   Are the inducing points solved for or are there a subset of the original ones?   How exactly is EQ 2 combined with the equation after line 148?  Question on Line 38: ``First, it can capture highly nonlinear interactions in the tensor, and is flexible enough to incorporate arbitrary subset of (meaningful) tensorial entries for the training.'' How is this different from other tensor factorizations? What does ``Highly nonlinear'' mean? How Gaussian priors allow more nonlinear interactions? These statements seem quite ad hoc to me.  Line 45: ``For example, one can choose a combination of balanced zero and nonzero elements to overcome the learning bias.'' But in the first paragraph of the introduction it is mentioned that zero elements are meaningless. Not sure what this sentence means.  The algorithmic complexity in 4.4 does not involve the number of iterations. Also, how do you invert K_BB, that should have at least p^3 complexity, right? I guess you just ignore this term?  There is way too much material in this paper. If it ends up resubmitted, I recommend that first the focus is on the formulation and detailed numerical experiments, and then on the distributed runs.   The comparison with linear logistic and svm is interesting, but one should compare with kernelized versions of those methods.


Edge-exchangeable graphs and sparsity
Many popular network models rely on the assumption of (vertex) exchangeability, in which the distribution of the graph is invariant to relabelings of the vertices. However, the Aldous-Hoover theorem guarantees that these graphs are dense or empty with probability one, whereas many real-world graphs are sparse. We present an alternative notion of exchangeability for random graphs, which we call edge exchangeability, in which the distribution of a graph sequence is invariant to the order of the edges. We demonstrate that edge-exchangeable models, unlike models that are traditionally vertex exchangeable, can exhibit sparsity. To do so, we outline a general framework for graph generative models; by contrast to the pioneering work of Caron and Fox [12], models within our framework are stationary across steps of the graph sequence. In particular, our model grows the graph by instantiating more latent atoms of a single random measure as the dataset size increases, rather than adding new atoms to the measure.

This paper introduces an alternative notion of exchangeability for random graphs, called edge exchangeability, by viewing a graph as a sequence of edges. In contrast to the classical Aldous-Hoover representation where the number of edges increases in proportional to the square of the number of vertices (a.s. dense or empty), the proposed graph representation is not only exchangeable in terms of edges but also leads to sparsity, i.e., the number of edges is asymptotically upper bounded by constant*#_vertex^2. The authors also propose the graph frequency model and simulate its generative process to validate the sparsity property.
[Technical quality] In Section 2.2, the authors have elaborated the step-augmented graph representation for edge-exchangeable graph sequences and try to define the edge exchangeability on the permutation of steps. But to me it still looks like directly applying the de Finetti's theorem to an exchangeable sequence conditioned on W Eq.(1), where the elements are pairs of vertices. Although W is two-dimensional and \theta has two indices, it actually is treated as a sequence (1-array), so the de Finetti's theorem can be directly applied. Is my understanding right? Or it is indeed a new notion of exchangeability?   The biggest flaw of this work may be that: According to the measure defined in Eq.(1), the generative process only generates a multi-set, whose elements are i.i.d. generated edges given W. However, a graph, if represented in edges, is a binary set. In Line 164, an additional operation is required to obtain the final graph by binarizing the multigraph. Thus the resulting binary graph is no longer edge-exchangeable and the measure define in Eq.(1) is not applicable to random binary graphs.  Can the rate w_{I,j} only be defined in the form of w_i w_j in Eq.(3)? All the discussions in the reminder of the paper are based on this definition. Any other form?  First mentioned in the introduction (Line 46) and some places later, the notion of “stationary” is not described or defined. Is it the same meaning defined in Section 7.8 [19]? And this “stationary” property is not further discussed or proved in the rest of the paper.  [Novelty] On one hand, the proposed edge-exchangeability is an application of the de Finetti's theorem to an exchangeable sequence conditioned on W, which is not novel.  On the other hand, the proposed graph frequency model tries to grow the number of vertices in the graph through increasing the number of steps; and tries to make connection to the well known Bayesian nonparametric model such as CRP by viewing Eq.(1) as DP and the posterior graph after n steps as CRP. Such interpretation and connection may be interesting.   [Impact] Graph representation for sparse graphs are very important and can be impactful.   [Presentation] The presentation and clarity of the paper is good. The running examples and connection to other forms of exchangeability in the supplementary material are helpful to understand the idea. Example 4.1 is not clearly described, and in Line 192, should it be Section 3?


Stochastic Online AUC Maximization
Area under ROC (AUC) is a metric which is widely used for measuring the classification performance for imbalanced data. It is of theoretical and practical interest to develop online learning algorithms that maximizes AUC for large-scale data. A specific challenge in developing online AUC maximization algorithm is that the learning objective function is usually defined over a pair of training examples of opposite classes, and existing methods achieves on-line processing with higher space and time complexity. In this work, we propose a new stochastic online algorithm for AUC maximization. In particular, we show that AUC optimization can be equivalently formulated as a convex-concave saddle point problem. From this saddle representation, a stochastic online algorithm (SOLAM) is proposed which has time and space complexity of one datum. We establish theoretical convergence of SOLAM with high probability and demonstrate its effectiveness on standard benchmark datasets.

This paper proposes an online algorithm for AUC maximization with low per-iteration space and time complexity.  The numerical experiments also suggest the performance of the proposed algorithm is comparable with existing algorithms with higher complexity.
1. I doubt whether the first equality of (8) holds:   In the first equality, the authors were assuming E[(wx)(wx') | y=1, y'=-1] = E[wx | y=1] E[wx' | y'=-1]. I guess the authors believe this is true because (x,y) and (x',y') are independent samples. However, two independent random variables are not necessarily conditional independent.   For instance, let X1 and X2 be two i.i.d. sample from distribution X= 1 with prob 1/2 and X=-1 with prob 1/2. Then, conditional on X1 = X2, they are no longer independent. Note E[X1 X2 | X1=X2] = 1 while E[X1 | X1 = X2] = E[X2| X1=X2] = 0.   Of course, my example above does not imply this paper could not have the independence for wx and wx' conditional on (y=1, y=-1). Maybe the authors have a different reason for the expectation identity, e.g., conditional uncorrelated? The authors should justify this step.  2. Related to the above, it is very surprising that the independence between (x,y) and (x', y') disappear in the saddle point function. Remember that the argument in (8) requires the independence of (x,y) and (x, y'). But in the end when everything is written as integration, this information is gone. The authors just treat two integration variables as the same thing! This is how they got equation (6). However, by doing so, the physical meaning of indicator functions inside the integration are changed. Some of the indicator functions should be defined in one label y and some should be defined in another label y'. (So when you run an online gradient type alg, you need two samples to evaluate the point gradient.) But this paper just define all indicator functions in terms of the same y at the end of the proof for Thm 1. That's why their alg only need one sample to have online update.   So I think even if (8) is correct, the authors can not simply write two expectations conditional on two different y as integrations with respect to the same variable. The original utility (4) by nature involves two samples and so should the reformulation in Thm 1.  3 Another comment on the proof of Thm 1 is:  The saddle point argument from equation (8) in the proof for Thm 1 can be better explained. If we define the objective function in (3) as h(w), then the proof starting from (8) is to show "by introducing other (dummy) variables, h(w) is equal to min_{a,b} max_{alpha} F(w,a,b, alpha) for any w." (Although, given w, the minimizing and maximizing points (a,b) and (alpha) are actually fixed,  F is preferred because expectations conditional on two labels are now decoupled as the sum of two expectations conditional on a single label.)  Now if we minimize w on both sides, min_w h(w) is equivalent to min_{w,a,b} max_{alpha} F(w,a,b, alpha) and the solutions with respect to w on both sides are the same. The solution w to min_{w,a,b} max_{alpha} F(w,a,b, alpha) is given by a saddle point because F is convex-concave function.