Modeling interactions between features improves the performance of machine learning solutions in many domains (e.g. recommender systems or sentiment analysis). In this paper, we introduce Exponential Machines (ExM), a predictor that models all interactions of every order. The key idea is to represent an exponentially large tensor of parameters in a factorized format called Tensor Train (TT). The Tensor Train format regularizes the model and lets you control the number of underlying parameters. To train the model, we develop a stochastic Riemannian optimization procedure, which allows us to fit tensors with 2 entries. We show that the model achieves state-of-the-art performance on synthetic data with highorder interactions and that it works on par with high-order factorization machines on a recommender system dataset MovieLens 100K.
1 INTRODUCTION
Machine learning problems with categorical data require modeling interactions between the features to solve them. As an example, consider a sentiment analysis problem – detecting whether a review is positive or negative – and the following dataset: ‘I liked it’, ‘I did not like it’, ‘I’m not sure’. Judging by the presence of the word ‘like’ or the word ‘not’ alone, it is hard to understand the tone of the review. But the presence of the pair of words ‘not’ and ‘like’ strongly indicates a negative opinion. If the dictionary has d words, modeling pairwise interactions requires O(d2) parameters and will probably overfit to the data. Taking into account all interactions (all pairs, triplets, etc. of words) requires impractical 2d parameters. In this paper, we show a scalable way to account for all interactions. Our contributions are:
• We propose a predictor that models all 2d interactions of d-dimensional data by representing the exponentially large tensor of parameters in a compact multilinear format – Tensor Train (TT-format) (Sec. 3). Factorizing the parameters into the TT-format leads to a better generalization, a linear with respect to d number of underlying parameters and inference time (Sec. 5). The TT-format lets you control the number of underlying parameters through the TT-rank – a generalization of the matrix rank to tensors.
• We develop a stochastic Riemannian optimization learning algorithm (Sec. 6.1). In our experiments, it outperformed the stochastic gradient descent baseline (Sec. 8.2) that is often used for models parametrized by a tensor decomposition (see related works, Sec. 9).
• We show that the linear model (e.g. logistic regression) is a special case of our model with the TT-rank equal 2 (Sec. 8.3).
• We extend the model to handle interactions between functions of the features, not just between the features themselves (Sec. 7).
2 LINEAR MODEL
In this section, we describe a generalization of a class of machine learning algorithms – the linear model. Let us fix a training dataset of pairs {(x(f), y(f))}Nf=1, where x(f) is a d-dimensional feature vector of f -th object, and y(f) is the corresponding target variable. Also fix a loss function `(ŷ, y) : R2 → R, which takes as input the predicted value ŷ and the ground truth value y. We call a model linear, if the prediction of the model depends on the features x only via the dot product between the features x and the d-dimensional vector of parameters w:
ŷlinear(x) = 〈x,w〉+ b, (1) where b ∈ R is the bias parameter. One of the approaches to learn the parameters w and b of the model is to minimize the following loss
N∑ f=1 ` ( 〈x(f),w〉+ b, y(f) ) + λ 2 ‖w‖22 , (2)
where λ is the regularization parameter. For the linear model we can choose any regularization term instead of L2, but later the choice of the regularization term will become important (see Sec. 6.1). Several machine learning algorithms can be viewed as a special case of the linear model with an appropriate choice of the loss function `(ŷ, y): least squares regression (squared loss), Support Vector Machine (hinge loss), and logistic regression (logistic loss).
3 OUR MODEL
Before introducing our model equation in the general case, consider a 3-dimensional example. The equation includes one term per each subset of features (each interaction)
ŷ(x) =W000 +W100 x1 +W010 x2 +W001x3 +W110 x1x2 +W101 x1x3 +W011 x2x3 +W111 x1x2x3.
(3)
Note that all permutations of features in a term (e.g. x1x2 and x2x1) correspond to a single term and have exactly one associated weight (e.g. W110). In the general case, we enumerate the subsets of features with a binary vector (i1, . . . , id), where ik = 1 if the k-th feature belongs to the subset. The model equation looks as follows
ŷ(x) = 1∑ i1=0 . . . 1∑ id=0 Wi1...id d∏ k=1 xikk . (4)
Here we assume that 00 = 1. The model is parametrized by a d-dimensional tensor W , which consist of 2d elements. The model equation (4) is linear with respect to the weight tensor W . To emphasize this fact and simplify the notation we rewrite the model equation (4) as a tensor dot product ŷ(x) = 〈X ,W〉, where the tensor X is defined as follows
Xi1...id = d∏ k=1 xikk . (5)
Note that there is no need in a separate bias term, since it is already included in the model as the weight tensor elementW0...0 (see the model equation example (3)). The key idea of our method is to compactly represent the exponentially large tensor of parameters W in the Tensor Train format (Oseledets, 2011).
4 TENSOR TRAIN
A d-dimensional tensor A is said to be represented in the Tensor Train (TT) format (Oseledets, 2011), if each of its elements can be computed as the following product of d− 2 matrices and 2 vectors
Ai1...id = G1[i1] . . . Gd[id], (6)
where for any k = 2, . . . , d − 1 and for any value of ik, Gk[ik] is an r × r matrix, G1[i1] is a 1 × r vector and Gd[id] is an r × 1 vector (see Fig. 1). We refer to the collection of matrices Gk corresponding to the same dimension k (technically, a 3-dimensional array) as the k-th TT-core, where k = 1, . . . , d. The size r of the slices Gk[ik] controls the trade-off between the representational power of the TT-format and computational efficiency of working with the tensor. We call r the TT-rank of the tensor A. An attractive property of the TT-format is the ability to perform algebraic operations on tensors without materializing them, i.e. by working with the TT-cores instead of the tensors themselves. The TT-format supports computing the norm of a tensor and the dot product between tensors; element-wise sum and element-wise product of two tensors (the result is a tensor in the TT-format with increased TT-rank), and some other operations (Oseledets, 2011).
5 INFERENCE
In this section, we return to the model proposed in Sec. 3 and show how to compute the model equation (4) in linear time. To avoid the exponential complexity, we represent the weight tensor W and the data tensor X (5) in the TT-format. The TT-ranks of these tensors determine the efficiency of the scheme. During the learning, we initialize and optimize the tensor W in the TT-format and explicitly control its TT-rank. The TT-rank of the tensor X always equals 1. Indeed, the following TT-cores give the exact representation of the tensor X
Gk[ik] = x ik k ∈ R 1×1, k = 1, . . . , d.
The k-th core Gk[ik] is a 1× 1 matrix for any value of ik ∈ {0, 1}, hence the TT-rank of the tensor X equals 1. Now that we have a TT-representations of tensors W and X , we can compute the model response ŷ(x) = 〈X ,W〉 in the linear time with respect to the number of features d. Theorem 1. The model response ŷ(x) can be computed in O(r2d), where r is the TT-rank of the weight tensor W .
We refer the reader to Appendix A where we propose an inference algorithm with O(r2d) complexity and thus prove Theorem 1. The TT-rank of the weight tensor W is a hyper-parameter of our method and it controls the efficiency vs. flexibility trade-off. A small TT-rank regularizes the model and yields fast learning and inference but restricts the possible values of the tensor W . A large TT-rank allows any value of the tensor W and effectively leaves us with the full polynomial model without any advantages of the TT-format.
6 LEARNING
Learning the parameters of the proposed model corresponds to minimizing the loss under the TT-rank constraint:
minimize W L(W), subject to TT-rank(W) = r0, (7)
where the loss is defined as follows
L(W) = N∑ f=1 ` ( 〈X (f),W〉, y(f) ) + λ 2 ‖W‖2F , ‖W‖ 2 F = 1∑ i1=0 . . . 1∑ id=0 W2i1...id . (8)
We consider two approaches to solving problem (7). In a baseline approach, we optimize the objective L(W) with stochastic gradient descent applied to the underlying parameters of the TT-format of the tensor W .
A simple alternative to the baseline is to perform gradient descent with respect to the tensor W , that is subtract the gradient from the current estimate of W on each iteration. The TT-format indeed allows to subtract tensors, but this operation increases the TT-rank on each iteration, making this approach impractical. To improve upon the baseline and avoid the TT-rank growth, we exploit the geometry of the set of tensors that satisfy the TT-rank constraint (7) to build a Riemannian optimization procedure (Sec. 6.1). We experimentally show the advantage of this approach over the baseline in Sec. 8.2.
6.1 RIEMANNIAN OPTIMIZATION
The set of all d-dimensional tensors with fixed TT-rank r
Mr = {W ∈ R2×...×2 : TT-rank(W) = r} forms a Riemannian manifold (Holtz et al., 2012). This observation allows us to use Riemannian optimization to solve problem (7). Riemannian gradient descent consists of the following steps which are repeated until convergence (see Fig. 2 for an illustration):
1. Project the gradient ∂L∂W on the tangent space ofMr taken at the point W . We denote the tangent space as TWMr and the projection as G = PTWMr ( ∂L∂W ). 2. Follow along G with some step α (this operation increases the TT-rank). 3. Retract the new point W − αG back to the manifoldMr, that is decrease its TT-rank to r.
We now describe how to implement each of the steps outlined above. Lubich et al. (2015) proposed an algorithm to project a TT-tensor Z on the tangent space of Mr at a point W which consists of two steps: preprocess W in O(dr3) and project Z in O(dr2 TT-rank(Z)2). Lubich et al. (2015) also showed that the TT-rank of the projection is bounded by a constant that is independent of the TT-rank of the tensor Z:
TT-rank(PTWMr (Z)) ≤ 2TT-rank(W) = 2r.
Let us consider the gradient of the loss function (8)
∂L ∂W = N∑ f=1 ∂` ∂ŷ X (f) + λW . (9)
Using the fact that PTWMr (W) = W and that the projection is a linear operator we get
PTWMr
( ∂L
∂W
) = N∑ f=1 ∂` ∂ŷ PTWMr (X (f)) + λW . (10)
Since the resulting expression is a weighted sum of projections of individual data tensors X (f), we can project them in parallel. Since the TT-rank of each of them equals 1 (see Sec. 5), allN projections cost O(dr2(r +N)) in total. The TT-rank of the projected gradient is less or equal to 2r regardless of the dataset size N . Note that here we used the particular choice of the regularization term. For terms other than L2 (e.g. L1), the gradient may have arbitrary large TT-rank. As a retraction – a way to return back to the manifoldMr – we use the TT-rounding procedure (Oseledets, 2011). For a given tensor W and rank r the TT-rounding procedure returns a tensor Ŵ = TT-round(W , r) such that its TT-rank equals r and the Frobenius norm of the residual ‖W−Ŵ‖F is as small as possible. The computational complexity of the TT-rounding procedure is O(dr3). Since we aim for big datasets, we use a stochastic version of the Riemannian gradient descent: on each iteration we sample a random mini-batch of objects from the dataset, compute the stochastic gradient for this mini-batch, make a step along the projection of the stochastic gradient, and retract back to the manifold (Alg. 1). An iteration of the stochastic Riemannian gradient descent consists of inferenceO(dr2M), projection O(dr2(r+M)), and retractionO(dr3), which yieldsO(dr2(r+M)) total computational complexity.
Algorithm 1 Riemannian optimization Input: Dataset {(x(f), y(f))}Nf=1, desired TT-rank r0, number of iterations T , mini-batch size M , learning rate α, regularization strength λ Output: W that approximately minimizes (7) Train linear model (2) to get the parameters w and b Initialize the tensor W0 from w and b with the TT-rank equal r0 for t := 1 to T do
Sample M indices h1, . . . , hM ∼ U({1, . . . , N}) Dt := ∑M j=1 ∂` ∂ŷX
(hj) + λWt−1 Gt := PTWt−1Mr (Dt) (10) Wt := TT-round(Wt−1 − αGt, r0)
end for
6.2 INITIALIZATION
We found that a random initialization for the TT-tensor W sometimes freezes the convergence of optimization method (Sec. 8.3). We propose to initialize the optimization from the solution of the corresponding linear model (1). The following theorem shows how to initialize the weight tensor W from a linear model. Theorem 2. For any d-dimensional vector w and a bias term b there exist a tensor W of TT-rank 2, such that for any d-dimensional vector x and the corresponding object-tensor X the dot products 〈x,w〉 and 〈X ,W〉 coincide.
The proof is provided in Appendix B.
7 EXTENDING THE MODEL
In this section, we extend the proposed model to handle polynomials of any functions of the features. As an example, consider the logarithms of the features in the 2-dimensional case:
ŷ log(x) =W00 +W01x1 +W10x2 +W11x1x2 +W20 log(x1) +W02 log(x2) +W12 x1 log(x2) +W21 x2 log(x1) +W22 log(x1) log(x2).
In the general case, to model interactions between ng functions g1, . . . , gng of the features we redefine the object-tensor as follows:
Xi1...id = d∏ k=1 c(xk, ik),
where
c(xk, ik) =  1, if ik = 0, g1(xk), if ik = 1, . . .
gng (xk), if ik = ng,
The weight tensor W and the object-tensor X are now consist of (ng + 1)d elements. After this change to the object-tensor X , learning and inference algorithms will stay unchanged compared to the original model (4).
Categorical features. Our basic model handles categorical features xk ∈ {1, . . . ,K} by converting them into one-hot vectors xk,1, . . . , xk,K . The downside of this approach is that it wastes the model capacity on modeling non-existing interactions between the one-hot vector elements xk,1, . . . , xk,K which correspond to the same categorical feature. Instead, we propose to use one TT-core per categorical feature and use the model extension technique with the following function
c(xk, ik) = { 1, if xk = ik or ik = 0, 0, otherwise.
This allows us to cut the number of parameters per categorical feature from 2Kr2 to (K + 1)r2 without losing any representational power.
8 EXPERIMENTS
We release a Python implementation of the proposed algorithm and the code to reproduce the experiments1. For the operations related to the TT-format, we used the TT-Toolbox2.
8.1 DATASETS
The datasets used in the experiments (see details in Appendix C)
1. UCI (Lichman, 2013) Car dataset is a classification problem with 1728 objects and 21 binary features (after one-hot encoding). We randomly splitted the data into 1382 training and 346 test objects and binarized the labels for simplicity.
2. UCI HIV dataset is a binary classification problem with 1625 objects and 160 features, which we randomly splitted into 1300 training and 325 test objects.
3. Synthetic data. We generated 100 000 train and 100 000 test objects with 30 features and set the ground truth target variable to a 6-degree polynomial of the features.
4. MovieLens 100K is a recommender system dataset with 943 users and 1682 movies (Harper & Konstan, 2015). We followed Blondel et al. (2016a) in preparing 2703 one-hot features and in turning the problem into binary classification.
8.2 RIEMANNIAN OPTIMIZATION
In this experiment, we compared two approaches to training the model: Riemannian optimization (Sec. 6.1) vs. the baseline (Sec. 6). In this and later experiments we tuned the learning rate of both Riemannian and SGD optimizers with respect to the training loss after 100 iterations by the grid search with logarithmic grid. On the Car and HIV datasets we turned off the regularization (λ = 0) and used rank r = 4. We report that on the Car dataset Riemannian optimization (learning rate α = 40) converges faster and achieves better final point than the baseline (learning rate α = 0.03) both in terms of the training and test losses (Fig. 3a, 5a). On the HIV dataset Riemannian optimization (learning rate α = 800) converges to the value 10−4 around 20 times faster than the baseline (learning rate α = 0.001, see Fig. 3b), but the model overfitts to the data (Fig. 5b). The results on the synthetic dataset with high-order interactions confirm the superiority of the Riemannian approach over SGD – we failed to train the model at all with SGD (Fig. 6). On the MovieLens 100K dataset, we have only used SGD-type algorithms, because using the one-hot feature encoding is much slower than using the categorical version (see Sec. 7), and we have yet to implement the support for categorical features for the Riemannian optimizer. On the bright side, prototyping the categorical version of ExM in TensorFlow allowed us to use a GPU accelerator.
8.3 INITIALIZATION
In this experiment, we compared random initialization with the initialization from the solution of the corresponding linear problem (Sec. 6.2). We explored two ways to randomly initialize a TT-tensor: 1) filling its TT-cores with independent Gaussian noise; 2) initializing W to represent a linear model with random coefficients (sampled from a standard Gaussian). We report that on the Car dataset type-1 random initialization slowed the convergence compared to initialization from the linear model solution (Fig. 3a), while on the HIV dataset the convergence was completely frozen (Fig. 3b). Two possible reasons for this effect are: a) the vanishing and exploding gradients problem (Bengio et al., 1994) that arises when dealing with a product of a large number of factors (160 in the case of the HIV dataset); b) initializing the model in such a way that high-order terms dominate we may force the gradient-based optimization to focus on high-order terms, while it may be more stable to start with low-order terms instead. Type-2 initialization (a random linear model) indeed worked on par with the best linear initialization on the Car, HIV, and synthetic datasets (Fig. 3b, 6).
1https://github.com/Bihaqo/exp-machines 2https://github.com/oseledets/ttpy
8.4 COMPARISON TO OTHER APPROACHES
On the synthetic dataset with high-order interactions we campared Exponential Machines (the proposed method) with scikit-learn implementation (Pedregosa et al., 2011) of logistic regression, random forest, and kernel SVM; FastFM implementation (Bayer, 2015) of 2-nd order Factorization Machines; our implementation of high-order Factorization Machines3; and a feed-forward neural network implemented in TensorFlow (Abadi et al., 2015). We used 6-th order FM with the Adam optimizer (Kingma & Ba, 2014) for which we’d chosen the best rank (20) and learning rate (0.003) based on the training loss after the first 50 iterations. We tried several feed-forward neural networks with ReLU activations and up to 4 fully-connected layers and 128 hidden units. We compared the models based on the Area Under the Curve (AUC) metric since it is applicable to all methods and is robust to unbalanced labels (Tbl. 1). On the MovieLens 100K dataset we used the categorical features representation described in Sec. 7. Our model obtained 0.784 test AUC with the TT-rank equal 10 in 273 seconds on a Tesla K40 GPU (the inference time is 0.3 seconds per 78800 test objects); our implentation of 3-rd order FM obtained 0.782; logistic regression obtained 0.782; and Blondel et al. (2016a) reported 0.786 with 3-rd order FM on the same data.
3https://github.com/geffy/tffm
8.5 TT-RANK
The TT-rank is one of the main hyperparameters of the proposed model. Two possible strategies can be used to choose it: grid-search or DMRG-like algorithms (see Sec. 9). In our experiments we opted for the former and observed that the model is fairly robust to the choice of the TT-rank (see Fig. 4), but a too small TT-rank can hurt the accuracy (see Tbl. 1).
9 RELATED WORK
Kernel SVM is a flexible non-linear predictor and, in particular, it can model interactions when used with the polynomial kernel (Boser et al., 1992). As a downside, it scales at least quadratically with the dataset size (Bordes et al., 2005) and overfits on highly sparse data. With this in mind, Rendle (2010) developed Factorization Machine (FM), a general predictor that models pairwise interactions. To overcome the problems of polynomial SVM, FM restricts the rank of the weight matrix, which leads to a linear number of parameters and generalizes better on sparse data. FM running time is linear with respect to the number of nonzero elements in the data, which allows scaling to billions of training entries on sparse problems. For high-order interactions FM uses CP-format (Caroll & Chang, 1970; Harshman, 1970) to represent the tensor of parameters. The choice of the tensor factorization is the main difference between the high-order FM and Exponential Machines. The TT-format comes with two advantages over the CP-format: first, the TT-format allows for Riemannian optimization; second, the problem of finding the best TT-rank r approximation to a given tensor always has a solution and can be solved in polynomial time. We found Riemannian optimization superior to the SGD baseline (Sec. 6) that was used in several other models parametrized by a tensor factorization (Rendle, 2010; Lebedev et al., 2014; Novikov et al., 2015). Note that CP-format also allows for Riemannian optimization, but only for 2-order tensors (and thereafter 2-order FM). A number of works used full-batch or stochastic Riemannian optimization for data processing tasks (Meyer et al., 2011; Tan et al., 2014; Xu & Ke, 2016; Zhang et al., 2016). The last work (Zhang et al., 2016) is especially interesting in the context of our method, since it improves the convergence rate of stochastic Riemannian gradient descent and is directly applicable to our learning procedure. In a concurrent work, Stoudenmire & Schwab (2016) proposed a model that is similar to ours but relies on the trigonometric basis (cos(π2x), sin( π 2x)) in contrast to polynomials (1, x) used in Exponential Machines (see Sec. 7 for an explanation on how to change the basis). They also proposed a different learning procedure inspired by the DMRG algorithm (Schollwöck, 2011), which allows to automatically choose the ranks of the model, but is hard to adapt to the stochastic regime. One of the possible ways to combine strengths of the DMRG and Riemannian approaches is to do a full DMRG sweep once in a few epochs of the stochastic Riemannian gradient descent to adjust the ranks. Other relevant works include the model that approximates the decision function with a multidimensional Fourier series whose coefficients lie in the TT-format (Wahls et al., 2014); and models that are similar to FM but include squares and other powers of the features: Tensor Machines (Yang & Gittens, 2015) and Polynomial Networks (Livni et al., 2014). Tensor Machines also enjoy a theoretical generalization bound. In another relevant work, Blondel et al. (2016b) boosted the efficiency of FM and Polynomial Networks by casting their training as a low-rank tensor estimation problem, thus making it multi-convex and allowing for efficient use of Alternative Least Squares types of algorithms. Note that Exponential Machines are inherently multi-convex.
10 DISCUSSION
We presented a predictor that models all interactions of every order. To regularize the model and to make the learning and inference feasible, we represented the exponentially large tensor of parameters in the Tensor Train format. To train the model, we used Riemannian optimization in the stochastic regime and report that it outperforms a popular baseline based on the stochastic gradient descent. However, the Riemannian learning algorithm does not support sparse data, so for dataset with hundreds of thousands of features we are forced to fall back on the baseline learning method. We found that training process is sensitive to initialization and proposed an initialization strategy based on the solution of the corresponding linear problem. The solutions developed in this paper for the stochastic Riemannian optimization may suit other machine learning models parametrized by tensors in the TT-format.
A PROOF OF THEOREM 1
Theorem 1 states that the inference complexity of the proposed algorithm is O(r2d), where r is the TT-rank of the weight tensor W . In this section, we propose an algorithm that achieve the stated complexity and thus prove the theorem.
Proof. Let us rewrite the definition of the model response (4) assuming that the weight tensor W is represented in the TT-format (6)
ŷ(x) = ∑
i1,...,id
Wi1...id ( d∏ k=1 xikk ) = ∑ i1,...,id G1[i1] . . . Gd[id] ( d∏ k=1 xikk ) .
Let us group the factors that depend on the variable ik, k = 1, . . . , d
ŷ(x) = ∑
i1,...,id
xi11 G1[i1] . . . x id d Gd[id] =
( 1∑
i1=0
xi11 G1[i1]
) . . . ( 1∑
id=0
xidd Gd[id] ) = A1︸︷︷︸
1×r A2︸︷︷︸ r×r . . . Ad︸︷︷︸ r×1 ,
where the matrices Ak for k = 1, . . . , d are defined as follows
Ak = 1∑ ik=0 xikk Gk[ik] = Gk[0] + xkGk[1].
The final value ŷ(x) can be computed from the matricesAk via d−1 matrix-by-vector multiplications and 1 vector-by-vector multiplication, which yields O(r2d) complexity. Note that the proof is constructive and corresponds to an implementation of the inference algorithm.
B PROOF OF THEOREM 2
Theorem 2 states that it is possible to initialize the weight tensor W of the proposed model from the weights w of the linear model. Theorem. For any d-dimensional vector w and a bias term b there exist a tensor W of TT-rank 2, such that for any d-dimensional vector x and the corresponding object-tensor X the dot products 〈x,w〉 and 〈X ,W〉 coincide.
To proof the theorem, in the rest of this section we show that the tensor W from Theorem 2 is representable in the TT-format with the following TT-cores
G1[0] = [ 1 0 ] , G1[1] = [ 0 w1 ] ,
Gd[0] = [ b 1 ] , Gd[1] = [ wd 0 ] ,
∀ 2 ≤ k ≤ d− 1
Gk[0] = [ 1 0 0 1 ] , Gk[1] = [ 0 wk 0 0 ] ,
(11)
and thus the TT-rank of the tensor W equals 2. We start the proof with the following lemma: Lemma 1. For the TT-cores (11) and any p = 1, . . . , d− 1 the following invariance holds:
G1[i1] . . . Gp[ip] =  [ 1 0 ] , if ∑p q=1 iq = 0, [ 0 0 ] , if ∑p q=1 iq ≥ 2, [ 0 wk ] , if ∑p q=1 iq = 1,
and ik = 1.
Proof. We prove the lemma by induction. Indeed, for p = 1 the statement of the lemma becomes
G1[i1] = { [ 1 0 ] , if i1 = 0, [ 0 w1 ] , if i1 = 1,
which holds by definition of the first TT-core G1[i1]. Now suppose that the statement of Lemma 1 is true for some p − 1 ≥ 1. If ip = 0, then Gp[ip] is an identity matrix and G1[i1] . . . Gp[ip] = G1[i1] . . . Gp−1[ip−1]. Also, ∑p q=1 iq = ∑p−1 q=1 iq , so the statement of the lemma stays the same. If ip = 1, then there are 3 options:
• If ∑p−1 q=1 iq = 0, then ∑p q=1 iq = 1 and
G1[i1] . . . Gp[ip] = [ 1 0 ]Gp[1] = [ 0 wp ] . • If ∑p−1 q=1 iq ≥ 2, then ∑p q=1 iq ≥ 2 and
G1[i1] . . . Gp[ip] = [ 0 0 ]Gp[1] = [ 0 0 ] . • If ∑p−1 q=1 iq = 1 with ik = 1, then ∑p q=1 iq ≥ 2 and
G1[i1] . . . Gp[ip] = [ 0 wk ]Gp[1] = [ 0 0 ] .
Which is exactly the statement of Lemma 1.
Proof of Theorem 2. The product of all TT-cores can be represented as a product of the first p = d−1 cores times the last core Gd[id] and by using Lemma 1 we get
Wi1...id = G1[i1] . . . Gd−1[id−1]Gd[id] =  b, if ∑d q=1 iq = 0, 0, if ∑d q=1 iq ≥ 2, wk, if ∑d q=1 iq = 1,
and ik = 1. The elements of the obtained tensor W that correspond to interactions of order ≥ 2 equal to zero; the weight that corresponds to xk equals to wk; and the bias termW0...0 = b. The TT-rank of the obtained tensor equal 2 since its TT-cores are of size 2× 2.
C DETAILED DESCRIPTION OF THE DATASETS
We used the following datasets for the experimental evaluation
1. UCI (Lichman, 2013) Car dataset is a classification problem with 1728 objects and 21 binary features (after one-hot encoding). We randomly splitted the data into 1382 training and 346 test objects. For simplicity, we binarized the labels: we picked the first class (‘unacc’) and made a one-versus-rest binary classification problem from the original Car dataset.
2. UCI (Lichman, 2013) HIV dataset is a binary classification problem with 1625 objects and 160 features, which we randomly splitted into 1300 training and 325 test objects.
3. Synthetic data. We generated 100 000 train and 100 000 test objects with 30 features. Each entry of the data matrix X was independently sampled from {−1,+1} with equal probabilities 0.5. We also uniformly sampled 20 subsets of features (interactions) of order 6: j11 , . . . , j 1 6 , . . . , j 20 1 , . . . , j 20 6 ∼ U{1, . . . , 30}. We set the ground truth target variable to a
deterministic function of the input: y(x) = ∑20 z=1 εz ∏6 h=1 xjzh , and sampled the weights of the interactions from the uniform distribution: ε1, . . . , ε20 ∼ U(−1, 1).
4. MovieLens 100K. MovieLens 100K is a recommender system dataset with 943 users and 1682 movies (Harper & Konstan, 2015). We followed Blondel et al. (2016a) in preparing the features and in turning the problem into binary classification. For users, we converted age (rounded to decades), living area (the first digit of the zipcode), gender and occupation into a binary indicator vector using one-hot encoding. For movies, we used the release year (rounded to decades) and genres, also encoded. This process yielded 49+29 = 78 additional one-hot features for each user-movie pair (943 + 1682 + 78 features in total). Original ratings were binarized using 5 as a threshold. This results in 21200 positive samples, half of which were used for traininig (with equal amount of sampled negative examples) and the rest were used for testing.

The paper presents an application of a tensor factorization to linear models, which allows to consider higher-order interactions between variables in classification (and regression) problems, and that maintains computational feasibility, being linear in the dimension. The factorization employed is based on the TT format, first proposed by Oseledests (2011). The authors also propose the adoption of a Riemannian optimization scheme to explicit consider the geometry of the tensor manifold, and thus speed up convergence.
The paper in general is well written, it presents an interesting application of the TT tensor format for linear models (together with an application of Riemannian optimization), which in my opinion is quite interesting since it has a wide range of possible applications in different algorithms in machine learning.
On the other side, I have some concerns are about the experimental part, which I consider not at the level of the rest of the paper, for instance in terms of number of experiments on real datasets, role of dropout in real datasets, comparison with other algorithms on real datasets. Moreover the authors do not take into account explicitly the problem of the choice of the rank to be used in the experiments. In general the experimental section seems a collection of preliminary experiments where different aspects have been tested by not in a organic way.
I think the paper is close to a weak acceptance / weak rejection, I don't rate it as a full acceptance paper, mainly due to the non-satisfactory experiment setting. In case of extra experiments confirming the goodness of the approach, I believe the paper could have much better scores.
Some minor comments:
-formula 2: Obvious comment: learning the parameters of the model in (1) can be done as in (2), but also in other ways, depending on the approach you are using.
-the fact that the rank is bounded by 2r, before formula 9, is explained in Lubich et al., 2015?
-after formula 10: why the N projections in total they cost O(dr^2(r+N)), it should be O(Ndr^2(r+1)), no? since each of the elements of the summation has rank 1, and the cost for each of them is O(dr^2(r+TT_rank(Z)^2)), where TT-rank(Z)=1. Am I wrong?
-section 6.2: can you explain why the random initialization freezes the convergence? This seems interesting but not motivated. Any guess?
-section 6.3: you adopt dropout: can you comment in particular on the advantages it gives in the context of the exponential machines? did you use it on real datasets?
-how do you choose r_0 in you experiments? with a validation set?
-in section 7: why you don't have x_1 x_2 among the variables?
-section 8: there is a typo in "experiments"
-section 8.1: "We simplicity, we binarized" I think there's a problem with the English language in this sentence
-section 8.3: "we report that dropout helps".. this is quite general statement, only tested on a synthetic dataset
-section 8.5: can you provide more results for this dataset, for instance in terms of training and inference time? or test wrt other algorithms?


We introduce a simple permutation equivariant layer for deep learning with set structure. This type of layer, obtained by parameter-sharing, has a simple implementation and linear-time complexity in the size of each set. We use deep permutation-invariant networks to perform point-could classification and MNISTdigit summation, where in both cases the output is invariant to permutations of the input. In a semi-supervised setting, where the goal is make predictions for each instance within a set, we demonstrate the usefulness of this type of layer in set-outlier detection as well as semi-supervised learning with clustering sideinformation.
1 INTRODUCTION
Recent progress in deep learning (LeCun et al., 2015) has witnessed its application to structured settings, including graphs (Bruna et al., 2013; Duvenaud et al., 2015) groups (Gens & Domingos, 2014; Christopher, 2014; Cohen & Welling, 2016), sequences and hierarchies (Irsoy & Cardie, 2014; Socher et al., 2013). Here, we introduce a simple permutation-equivariant layer for deep learning with set structure, where the primary dataset is a collection of sets, possibly of different sizes. Note that each instance may have a structure of its own, such as graph, image, or another set. In typical machine-learning applications, iid assumption implies that the entire data-set itself has a set structure. Therefore, our special treatment of the set structure is only necessary due to multiplicity of distinct yet homogeneous (data)sets. Here, we show that a simple parameter-sharing scheme enables a general treatment of sets within supervised and semi-supervised settings.
In the following, after introducing the set layer in Section 2, we explore several novel applications. Section 3 studies supervised learning with sets that requires “invariance” to permutation of inputs. Section 3.1 considers the task of summing multiple MNIST digits, and Section 3.2 studies an important application of sets in representing low-dimensional point-clouds. Here, we show that deep networks can successfully classify objects using their point-cloud representation.
Section 4 presents numerical study in semi-supervised setting where the output of the multi-layer network is “equivariant” to the permutation of inputs. We use permutation-equivariant layer to perform outlier detection on CelebA face dataset in Section 4.1 and improve galaxy red-shift estimates using its clustering information in Section 5.
2 PERMUTATION-EQUIVARIANT LAYER
Let xn ∈ X denote use x = [x1, . . . , xN ] to denote a vector of xn instances. Here, xn, could be a feature-vector, an image or any other structured object. Our goal is to design neural network layers that are “indifferent” to permutations of instances in x. Achieving this goal amounts to treating x as a “set” rather than a vector.
The function f : XN → YN is equivariant to the permutation of its inputs iff f(πx) = πf(x) ∀π ∈ SN
where the symmetric group SN is the set of all permutation of indices 1, . . . , N . Similarly, the function f : <N → < is invariant to permutation of its inputs –i.e., a.k.a. a symmetric function (David
et al., 1966)– iff f(πx) = f(x) ∀π ∈ SN . Here, the action of π on a vector x ∈ <N can be represented by a permutation matrix. With some abuse of notation, we use π ∈ {0, 1}N×N to also denote this matrix.
Given two permutation equivariance function f : XN → YN and g : YN → ZN , their composition is also permutation-equivariance; this is because g(f(πx)) = g(πf(x)) = πg(f(x)).
Consider the standard neural network layer
fΘ(x) . = σ(Θx) Θ ∈ <N×N (1)
where Θ is the weight vector and σ : < → < is a nonlinearity such as sigmoid function. σ : <N → <N is the point-wise application of σ to its input vector. The following theorem states the necessary and sufficient conditions for permutation-equivariance in this type of function. Theorem 2.1. The function fΘ : <N → <N as defined in Eq. (1) is permutation equivariant iff all the off-diagonal elements of Θ are tied together and all the diagonal elements are equal as well,
Θ = λI + γ (11T) λ, γ ∈ < 1 = [1, . . . , 1]T ∈ <N (2) where I is the identity matrix.1
This function is simply a non-linearity applied to a weighted combination of I) its input Ix and; II) the sum of input values (11T)x. Since summation does not depend on the permutation, the layer is permutation-equivariant. Therefore we can manipulate the operations and parameters in this layer, for example to get another variation
f(x) . = σ(λIx− γ(max
n x)1) (3)
where the max operation over elements of the set is (similar to summation) commutative and using −γ instead of +γ amounts to a reparametrization. In practice using this variation performs better in some applications. This may be due to the fact that for λ = γ, the input to the non-linearity is max-normalized.
For multiple input-output channels, we may speed up the operation of the layer using matrix multiplication. Suppose we haveK input channels –corresponding toK features for each instance in the set– with a set of sizeN , andK ′ output channels. Here, x ∈ <N×K and fΘ : <N×K → <N×K ′ . The permutation-equivariant layer parameters are Λ,Γ ∈ <K,K′ (replacing λ and γ in Eq. (2)). The output of this layer becomes
y = σ ( xΛ + 11TxΓ ) Similarly, the multiple input-output channel variation of the Eq. (3) is
y = σ ( xΛ − 1xmaxΓ ) (4)
where xmax = (maxn x) ∈ <1×K is a row-vector of maximum value of x ∈ <N×K over the “set” dimension. In practice, we may further reduce the number of parameters in favor of better generalization by factoring Γ and Λ and keeping a single λ ∈ <K,K′
y = σ ( β + ( x − 1(max n x) ) Γ )
(5)
1See the Appendix A for the proof.
where β ∈ <K′ is a bias parameter. This final variation of permutation-equivariant layer is simply a fully connected layer where input features are max-normalized within each set.
With multiple input-output channels, the complexity of this layer for each set is O(N K K ′). Subtracting the mean or max over the set also reduces the internal covariate shift (Ioffe & Szegedy, 2015) and we observe that for deep networks (even using tanh activation), batch-normalization is not required.
When applying dropout (Srivastava et al., 2014) to regularize permutation-equivariant layers with multiple output channels, it is often beneficial to simultaneously dropout the channels for all instances within a set. In particular, when set-members share similar features, independent dropout effectively does not regularize the model as the the network learns to replace the missing features from other set-members.
In the remainder of this paper, we demonstrate how this simple treatment of sets can solve novel and non-trivial problems that occasionally have no alternative working solutions within deep learning.
3 SUPERVISED LEARNING
The permutation-equivariant layers that we introduced so far are useful for semi-supervised learning (or transductive) setting, where we intend to predict a value per each instance in every set. In supervised (or inductive) setting the task is to make a prediction for each set (rather than instances within them), and we require permutation “invariance” of f : XN → Y . A pooling operation over the set-dimension can turn any permutation equivariant function f : XN → YN , permutation invariant f(x) . = ⊕
n f(x). Here ⊕ is any commutative operation such as summation or maximization. In a related work Chen et al. (2014) construct deep permutation invariant features by pairwise coupling of features at the previous layer, where fi,j([xi, xj ]) . = [|xi − xj |, xi + xj ] is invariant to transposition of i and j.2
As we see shortly in Section 3.1, in the supervised setting, even simple application of setpooling, without max-normalization of Eq. (5) performs very well in practice. However, in semisupervised setting since there is no pooling operation, the permutation invariant layer requires maxnormalization in order to obtain the required information about the context of each instance.
3.1 PREDICTING THE SUM OF MNIST DIGITS
MNIST dataset (LeCun et al., 1998) contains 70,000 instances of 28×28 grey-scale stamps of digits in {0, . . . , 9}. We randomly sample a subset of N images from this dataset to build 10,000 “sets” of training and 10,000 sets of validation images, where the set-label is the sum of digits in that set (i.e., individual labels per image is unavailable).
2Due to change in the number of features/channels in each layer this approach cannot produce permutation“equivariant” layers. Also, this method requires a graph to guide the multi-resolution partitioning of the nodes, which is then used to define pairing of features in each layer.
In our first experiment, each set contains N = 3 images, and the set label is a number between 0 ≤ y ≤ 3 ∗ 9 = 27. We then considered four different models for predicting the sum:
I) Concatenating instances along the horizontal axis. II) Stacking images in each set as different input channels. III) Using set-pooling without max-normalization. IV) Using the permutation equivariant layer of Eq. (5) with set-pooling.
All models are defined to have similar number of layers and parameters; see Appendix B.1 for details. The output of all models is a (9N + 1)-way softmax, predicting the sum of N digits.
Figure 1(left) show the prediction accuracy over the validation-set for different models for N = 3. We see that using the set-layer performs the best. However, interestingly, using set-pooling alone produces similarly good results. We also observe that concatenating the digits eventually performs well, despite its lack of invariance. This is because due to the sufficiently large size of the dataset most permutations of length three appear in the training set.
However, as we increase the size of each set to N = 6, permutation invariance becomes crucial; see Figure 1(right). We see that using the default dropout rate of 20%, the model simply memorizes the input instances (indicated by discrepancy of training/validation error) and by increasing this dropout rate, the model simply predicts values close to the mean value. However, the permutation-invariant layer learns to predict the sum of six digits with> 80% accuracy, without having access to individual image labels. Performance using set-pooling alone is similar.
3.2 POINT CLOUD CLASSIFICATION
A low-dimensional point-cloud is a set of low-dimensional vectors. This type of data is frequently encountered in various applications from robotics and vision to cosmology. In these applications, point-cloud data is often converted to voxel or mesh representation at a preprocessing step (e.g., Maturana & Scherer, 2015; Ravanbakhsh et al., 2016; Lin et al., 2004). Since the output of many range sensors such as LiDAR – which are extensively used in applications such as autonomous vehicles – is in the form of point-cloud, direct application of deep learning methods to point-cloud is highly desirable. Moreover, when working with point-clouds rather than voxelized 3D objects, it is easy to apply transformations such as rotation and translation as differentiable layers at low cost.
Here, we show that treating the point-cloud data as a set, we can use the set-equivariant layer of Eq. (5) to classify point-cloud representation of a subset of ShapeNet objects (Chang et al., 2015), called ModelNet40 (Wu et al., 2015). This subset consists of 3D representation of 9,843 training and 2,468 test instances belonging to 40 classes of objects; see Fig. 2. We produce point-clouds with 100, 1000 and 5000 particles each (x, y, z-coordinates) from the mesh representation of objects using the point-cloud-library’s sampling routine (Rusu & Cousins, 2011). Each set is normalized by the initial layer of the deep network to have zero mean (along individual axes) and unit (global)
variance. Additionally we experiment with the K-nearest neighbor graph of each point-cloud and report the results using graph-convolution; see Appendix B.3 for model details.
Table 1 compares our method against the competition.3 Note that we achieve our best accuracy using 5000×3 dimensional representation of each object, which is much smaller than most other methods. All other techniques use either voxelization or multiple view of the 3D object for classification. Interestingly, variations of view/angle-pooling (e.g., Su et al., 2015; Shi et al., 2015) can be interpreted as set-pooling where the class-label is invariant to permutation of different views. The results also shows that using fully-connected layers with set-pooling alone (without max-normalization over the set) works relatively well.
We see that reducing the number of particles to only 100, still produces comparatively good results. Using graph-convolution is computationally more challenging and produces inferior results in this setting. The results using 5000 particles is also invariant to small changes in scale and rotation around the z-axis; see Appendix B.3 for details.
Features. To visualize the features learned by the set layers, we used Adamax (Kingma & Ba, 2014) to locate 1000 particle coordinates maximizing the activation of each unit.4 Activating the tanh units beyond the second layer proved to be difficult. Figure 3 shows the particle-cloud-features learned at the first and second layers of our deep network. We observed that the first layer learns simple localized (often cubic) point-clouds at different (x, y, z) locations, while the second layer learns more complex surfaces with different scales and orientations.
4 SEMI-SUPERVISED LEARNING
In semi-supervised or transductive learning, some/all instances within each training set are labelled. Our goal is to make predictions for individual instances within a test set. Therefore, the permutation equivariant layer leverages the interaction between the set-members to label individual member. Note that in this case, we do not perform any pooling operation over the set dimension of the data.
4.1 SET ANOMALY DETECTION
The objective here is for the deep model to find the anomalous face in each set, simply by observing examples and without any access to the attribute values. CelebA dataset (Liu et al., 2015) contains
3The error-bar on our results is due to variations depending on the choice of particles during test time and it is estimated over three trials.
4We started from uniformly distributed set of particles and used a learning rate of .01 for Adamax, with first and second order moment of .1 and .9 respectively. We optimized the input in 105 iterations. The results of Fig. 3 are limited to instances where tanh units were successfully activated. Since the input at the first layer of our deep network is normalized to have a zero mean and unit standard deviation, we do not need to constrain the input while maximizing unit’s activation.
202,599 face images, each annotated with 40 boolean attributes. We use 64 × 64 stamps and using these attributes we build 18,000 sets, each containingN = 16 images (on the training set) as follows: after randomly selecting two attributes, we draw 15 images where those attributes are present and a single image where both attributes are absent. Using a similar procedure we build sets on the test images. No individual person’s face appears in both train and test sets.
Our deep neural network consists of 9 2D-convolution and max-pooling layers followed by 3 permutation-equivariant layers and finally a softmax layer that assigns a probability value to each set member (Note that one could identify arbitrary number of outliers using a sigmoid activation at the output.) Our trained model successfully finds the anomalous face in 75% of test sets. Visually inspecting these instances suggests that the task is non-trivial even for humans; see Fig. 4. For details of the model, training and more identification examples see Appendix B.2.
As a baseline, we repeat the same experiment by using a set-pooling layer after convolution layers, and replacing the permutation-equivariant layers with fully connected layers, with the same number of hidden units/output-channels, where the final layer is a 16-way softmax. The resulting network shares the convolution filters for all instances within all sets, however the input to the softmax is not equivariant to the permutation of input images. Permutation equivariance seems to be crucial here as the baseline model achieves a training and test accuracy of∼ 6.3%; the same as random selection.
5 IMPROVED RED-SHIFT ESTIMATION USING CLUSTERING INFORMATION
An important regression problem in cosmology is to estimate the red-shift of galaxies, corresponding to their age as well as their distance from us (Binney & Merrifield, 1998). Two common types of observation for distant galaxies include a) photometric and b) spectroscopic observations, where the latter can produce more accurate red-shift estimates.
One way to estimate the red-shift from photometric observations is using a regression model (Connolly et al., 1995). We use a multi-layer Perceptron for this purpose and use the more accurate spectroscopic red-shift estimates as the ground-truth. As another baseline, we have a photometric redshift estimate that is provided by the catalogue and uses various observations (including clustering information) to estimate individual galaxy-red-shift. Our objective is to use clustering information of the galaxies to improve our red-shift prediction using the multi-layer Preceptron.
Note that the prediction for each galaxy does not change by permuting the members of the galaxy cluster. Therefore, we can treat each galaxy cluster as a “set” and use permutation-equivariant layer to estimate the individual galaxy red-shifts.
For each galaxy, we have 17 photometric features 5 from the redMaPPer galaxy cluster catalog (Rozo & Rykoff, 2014), which contains photometric readings for 26,111 red galaxy clusters. In this task in contrast to the previous ones, sets have different cardinalities; each galaxy-cluster in this catalog has between ∼ 20− 300 galaxies – i.e., x ∈ <N(c)×17, where N(c) is the cluster-size. See Fig. 5(a) for distribution of cluster sizes. The catalog also provides accurate spectroscopic red-shift estimates for a subset of these galaxies as well as photometric estimates that uses clustering information. Fig. 5(b) reports the distribution of available spectroscopic red-shift estimates per cluster.
We randomly split the data into 90% training and 10% test clusters, and use the following simple architecture for semi-supervised learning. We use four permutation-equivariant layers with 128, 128, 128 and 1 output channels respectively, where the output of the last layer is used as red-shift estimate. The squared loss of the prediction for available spectroscopic red-shifts is minimized.6 Fig. 5(c) shows the agreement of our estimates with spectroscopic readings on the galaxies in the test-set with spectroscopic readings. The figure also compares the photometric estimates provided by the catalogue (see Rozo & Rykoff, 2014), to the ground-truth. As it is customary in cosmology literature, we report the average scatter |zspec−z|1+zspec , where zspec is the accurate spectroscopic measurement and z is a photometric estimate. The average scatter using our model is .023 compared to the scatter of .025 in the original photometric estimates for the redMaPPer catalog. Both of these values are averaged over all the galaxies with spectroscopic measurements in the test-set.
We repeat this experiment, replacing the permutation-equivariant layers with fully connected layers (with the same number of parameters) and only use the individual galaxies with available spectroscopic estimate for training. The resulting average scatter for multi-layer Perceptron is .026, demonstrating that using clustering information indeed improves photometric red-shift estimates.
CONCLUSION
We introduced a simple parameter-sharing scheme to effectively achieve permutation-equivariance in deep networks and demonstrated its effectiveness in several novel supervised and semi-supervised tasks. Our treatment of set structure also generalizes various settings in multi-instance learning (Ray et al., 2011; Zhou et al., 2009). In addition to our experimental settings, the permutation-invariant layer can be used for distribution regression and classification which have become popular recently (Szabo et al., 2016). In our experiments with point-cloud data we observed the model to be robust to the variations in the number of particles in each cloud, suggesting the usefulness of our method in the general setting of distribution regression – where the number of samples should not qualitatively affect our representation of a distribution. We leave further investigation of this direction to future work.
5We have a single measurement for each u,g,r, i and z band as well as measurement error bars, location of the galaxy in the sky, as well as the probability of each galaxy being the cluster center. We do not include the information regarding the richness estimates of the clusters from the catalog, for any of the methods, so that baseline multi-layer Preceptron is blind to the clusters.
6We use mini-batches of size 128, Adam (Kingma & Ba, 2014), with learning rate of .001, β1 = .9 and β2 = .999. All layers except for the last layer use Tanh units and simultaneous dropout with 50% dropout rate.
ACKNOWLEDGEMENT
We would like to thank Francois Lanusse for the pointing us to the redMaPPer dataset and the anonymous reviewers as well as Andrew Wagner for valuable feedback.
A PROOFS
Proof. of the Theorem 2.1 From definition of permutation equivariance fΘ(πx) = πfΘ(x) and definition of f in Eq. (1), the condition becomes σ(Θπx) = πσ(Θx), which (assuming sigmoid is a bijection) is equivalent to Θπ = πΘ. Therefore we need to show that the necessary and sufficient conditions for the matrix Θ ∈ <N×N to commute with all permutation matrices π ∈ Sn is given by Eq. (2). We prove this in both directions:
• To see why Θ = λI + γ (11T) commutes with any permutation matrix, first note that commutativity is linear – that is
Θ1π = πΘ1 ∧Θ2π = πΘ2 ⇒ (aΘ1 + bΘ2)π = π(aΘ1 + bΘ2).
Since both Identity matrix I, and constant matrix 11T, commute with any permutation matrix, so does their linear combination Θ = λI + γ (11T).
• We need to show that in a matrix Θ that commutes with “all” permutation matrices
– All diagonal elements are identical: Let πk,l for 1 ≤ k, l ≤ N, k 6= l, be a transposition (i.e., a permutation that only swaps two elements). The inverse permutation matrix of πk,l is the permutation matrix of πl,k = πTk,l. We see that commutativity of Θ with the transposition πk,l implies that Θk,k = Θl,l:
πk,lΘ = Θπk,l ⇒ πk,lΘπl,k = Θ ⇒ (πk,lΘπl,k)l,l = Θl,l ⇒ Θk,k = Θl,l Therefore, π and Θ commute for any permutation π, they also commute for any transposition πk,l and therefore Θi,i = λ ∀i. – All off-diagonal elements are identical: We show that since Θ commutes with any product of transpositions, any choice two off-diagonal elements should be identical. Let (i, j) and (i′, j′) be the index of two off-diagonal elements (i.e., i 6= j and i′ 6= j′). Moreover for now assume i 6= i′ and j 6= j′. Application of the transposition πi,i′Θ, swaps the rows i, i′ in Θ. Similarly, Θπj,j′ switches the jth column with j′th column. From commutativity property of Θ and π ∈ Sn we have
πj′,jπi,i′Θ = Θπj′,jπi,i′ ⇒ πj′,jπi,i′Θ(πj′,jπi,i′)−1 = Θ ⇒ πj′,jπi,i′Θπi′,iπj,j′ = Θ ⇒ (πj′,jπi,i′Θπi′,iπj,j′)i,j = Θi,j ⇒ Θi′,j′ = Θi,j
where in the last step we used our assumptions that i 6= i′, j 6= j′, i 6= j and i′ 6= j′. In the cases where either i = i′ or j = j′, we can use the above to show that Θi,j = Θi′′,j′′ and Θi′,j′ = Θi′′,j′′ , for some i′′ 6= i, i′ and j′′ 6= j, j′, and conclude Θi,j = Θi′,j′ .
B DETAILS OF MODELS
In the following, all our implementations use Tensorflow (Abadi et al., 2016).
B.1 MNIST SUMMATION
All nonlinearities are exponential linear units (ELU Clevert et al., 2015). All models have 4 convolution layers followed by max-pooling. The convolution layers have respectively 16-32-64-128 output channels and 5× 5 receptive fields. Each pooling, fully-connected and set-layer is followed by a 20% dropout. For models III and IV we use simultaneous dropout. In models I and II, the convolution layers are followed by two fullyconnected layers with 128 hidden units. In model III, after the first fully connected layer we perform
set-pooling followed by another dense layer with 128 hidden units. In the model IV, the convolution layers are followed by a permutation-equivariant layer with 128 output channels, followed by setpooling and a fully connected layer with 128 hidden units. For optimization, we used a learning rate of .0003 with Adam using the default β1 = .9 and β2 = .999.
B.2 FACE OUTLIER DETECTION MODEL
Our model has 9 convolution layers with 3 × 3 receptive fields. The model has convolution layers with 32, 32, 64 feature-maps followed by max-pooling followed by 2D convolution layers with 64, 64, 128 feature-maps followed by another max-pooling layer. The final set of convolution layers have 128, 128, 256 feature-maps, followed by a max-pooling layer with pool-size of 5 that reduces the output dimension to batch− size.N × 256, where the set-size N = 16. This is then forwarded to three permutation-equivariant layers with 256, 128 and 1 output channels. The output of final layer is fed to the Softmax, to identify the outlier. We use exponential linear units (Clevert et al., 2015), drop out with 20% dropout rate at convolutional layers and 50% dropout rate at the first two set layers. When applied to set layers, the selected feature (channel) is simultaneously dropped in all the set members of that particular set. We use Adam (Kingma & Ba, 2014) for optimization and use batch-normalization only in the convolutional layers. We use mini-batches of 8 sets, for a total of 128 images per batch.
B.3 MODELS FOR POINT-CLOUDS CLASSIFICATION
Set convolution. We use a network comprising of 3 permutation-equivariant layers with 256 channels followed by max-pooling over the set structure. The resulting vector representation of the set is then fed to a fully connected layer with 256 units followed by a 40-way softmax unit. We use Tanh activation at all layers and dropout on the layers after set-max-pooling (i.e., two dropout operations) with 50% dropout rate. Applying dropout to permutation-equivariant layers for point-cloud data deteriorated the performance. We observed that using different types of permutation-equivariant layers (see Section 2) and as few as 64 channels for set layers changes the result by less than 5% in classification accuracy.
For the setting with 5000 particles, we increase the number of units to 512 in all layers and randomly rotate the input around the z-axis. We also randomly scale the point-cloud by s ∼ U(.8, 1./.8). For this setting only, we use Adamax (Kingma & Ba, 2014) instead of Adam and reduce learning rate from .001 to .0005.
Graph convolution. For each point-cloud instance with 1000 particles, we build a sparse K-nearest neighbor graph and use the three point coordinates as input features. We normalized all graphs at the preprocessing step. For direct comparison with set layer, we use the exact architecture of 3 graphconvolution layer followed by set-pooling (global graph pooling) and dense layer with 256 units. We use exponential linear activation function instead of Tanh as it performs better for graphs. Due to over-fitting, we use a heavy dropout of 50% after graph-convolution and dense layers. Similar to dropout for sets, all the randomly selected features are simultaneously dropped across the graph nodes. the We use a mini-batch size of 64 and Adam for optimization where the learning rate is .001 (the same as that of permutation-equivariant counter-part).
Despite our efficient sparse implementation using Tensorflow, graph-convolution is significantly slower than the set layer. This prevented a thorough search for hyper-parameters and it is quite possible that better hyper-parameter tuning would improve the results that we report here.

This review is only an informed guess - unfortunately I cannot assess the paper due to my lack of understanding of the paper.
I have spent several hours trying to read this paper - but it has not been possible for me to follow - partially due to my own limitations, but also I think due to an overly abstract level of presentation. The paper is clearly written, but in the same way that a N. Bourbaki book is clearly written.
I would prefer to leave the accept/reject decision to the other reviewers who may have a better understanding - even if the authors had made a serious mistake, I would not be able to tell. My proposal is positive because the paper is apparently clearly written and the empirical evaluation is quite promising. But some effort will be needed in order to address the broader audience that could potentially be interested in the topic.
I therefore would like to provide feedback only at the level of presentation.
My main source of problems is that the authors do not try to ground their abstract formalism with concrete examples; when the examples show up it is by "revelation" rather than by explaining how they connect to the previous concepts.
The one example that could unlock most people's understanding is how convolution, or inner product operations connect with the setting described here. For what I know convolution is tied with space (or time) and is understood as an equivariant operation - shifting the signal shifts the output.
It is not explained how the '(x, x')' pairs used by the authors in order to build relations, structures and then to define invariance relate to this setting.
Going from sets, to relations, to functions, to operators, and then to shift-invariant operators (convolutions) involves many steps, and some hand-holding is needed.
Why is the 3x3 convolution associated to 9 relations?
Are these relations referring to the input at a given coordinate and its contribution to the output? (w_{offset} x_{i-offset})? In that case, why is there a backward arrow from the center node to the other nodes? And why are there arrows across nodes?
What is a Cardinal and what is a Cartesian convolution in signal processing terms? (clearly these are not standard terms).
Are we talking about separable filters?
What are the X and Square symbols in Figure 2? And what are the horizontal and vertical sub-graphs standing for? What is x_1 and what is x_{11},x_{1,2},x_{1,3} and what is the relationship between them?
I realize that to the authors these questions may seem to be trivial and left as  homework for the reader. But I think part of publishing a paper is doing a big part of the homework for the readers so that it becomes easy to get the idea.
Clearly the authors target the more general case - but spending some time to explain how the particular case is an instance of the the general case would be a good use of space.
I would propose that the authors explain what are  x, x_{I}, and x_{S} for the simplest possible example, e.g. convolving a 1x5 signal with a 1x3 filter, how the convolution filter parameters show up in the function f, as well as how the spatial invariance (or, equivariance) of convolution is reflected here.