Sparse coding represents a signal by a linear combination of only a few atoms of a learned over-complete dictionary. While sparse coding exhibits compelling performance for various machine learning tasks, the process of obtaining sparse code with fixed dictionary is independent for each data point without considering the geometric information and manifold structure of the entire data. We propose Support Regularized Sparse Coding (SRSC) which produces sparse codes that account for the manifold structure of the data by encouraging nearby data in the manifold to choose similar dictionary atoms. In this way, the obtained support regularized sparse codes capture the locally linear structure of the data manifold and enjoy robustness to data noise. We present the optimization algorithm of SRSC with theoretical guarantee for the optimization over the sparse codes. We also propose a feed-forward neural network termed Deep Support Regularized Sparse Coding (Deep-SRSC) as a fast encoder to approximate the sparse codes generated by SRSC. Extensive experimental results demonstrate the effectiveness of SRSC and Deep-SRSC.
1 INTRODUCTION
The aim of sparse coding is to represent an input vector by a linear combination of a few atoms of a learned dictionary which is usually over-complete, and the coefficients for the atoms are called sparse code. Sparse coding is widely applied in machine learning and signal processing, and sparse code is extensively used as a discriminative and robust feature representation with convincing performance for classification and clustering (Yang et al., 2009; Cheng et al., 2013; Zhang et al., 2013). Suppose the data X = [x1,x2, . . . ,xn] ∈ IRd×n lie in the d-dimensional Euclidean space IRd, and the dictionary matrix is D = [D1,D2, . . . ,Dp] ∈ IRd×p with each Dk ∈ IRd (k = 1, . . . , p) being an atom of the dictionary, sparse coding method seeks for the linear sparse representation with respect to the dictionary D for each vector x ∈X by solving the following convex optimization problem:
min D,Z n∑ i=1 1 2 ‖xi −DZi‖22 + λ‖Zi‖1 s.t. ‖Dk‖2 ≤ c0, k = 1, . . . , p
where λ is a weighting parameter for the `1-norm of z, and c0 is a positive constant that bounds the `2-norm of each dictionary atom. In (Gregor & LeCun, 2010), a feed-forward neural network named Learned Iterative Shrinkage and Thresholding Algorithm (LISTA) is proposed to produce the approximation for sparse coding (1). The architecture of LISTA is illustrated in Figure 1. The LISTA network involves an finite number of stages wherein each stage performs the following operation on the intermediate sparse code:
z(k+1) = hθ(Wx + Sz (k)), z(0) = 0 (1)
This material is based upon work supported by the National Science Foundation under Grant No. 1318971. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
where hθ is an element-wise shrinkage function defined as
[hθ(u)]k = sign(uk)(|uk| − θ)+, k = 1, . . . , p (2)
and (·)+ = max{·, 0} is the positive part of a number. Let f indicate the LISTA network and it generates the approximate sparse code z = f(x,Θ), where Θ = (W,S, θ) collectively denotes the parameters of the LISTA network. Suppose the optimal sparse codes for the training data x1, . . . ,xm are Z∗1, . . . ,Z∗m, then the parameters Θ are learned by minimizing the cost function which measures the distance between the predicted approximate sparse codes and the optimal sparse
codes: L(Θ) = 1m m∑ i=1 ‖Z∗i − f(xi,Θ)‖22. The optimization is performed by stochastic gradient descent and back-propagation. Inspired by LISTA, a series of previous works have designed neural networks to simulate different forms of linear coding and achieve end-to-end training for different tasks such as image super-resolution (Liu et al., 2016) and hashing (Wang et al., 2016).
Sparse coding is widely used to model high-dimensional data. Based on the formulation of sparse coding (1), it can be observed that the sparse code of each data point is obtained independently when the dictionary is fixed, which ignores the geometric information and manifold structure of the high-dimensional data. In order to obtain the sparse codes that account for the geometric information and manifold structure of the data, many regularized sparse coding methods, such as (Liu et al., 2010; He et al., 2011; Zheng et al., 2011; Gao et al., 2013), employ manifold assumption (Belkin et al., 2006). Manifold assumption in these methods imposes local smoothness on the sparse codes of nearby data, namely nearby data are encouraged to have similar sparse codes in the sense of `2-distance, and they are termed `2-Regularized Sparse Coding (`2-RSC). In this paper, we propose Support Regularized Sparse Coding (SRSC). Compared to `2-RSC, SRSC captures the locally linear structure of the data manifold by encouraging nearby data to share dictionary atoms. In addition, SRSC enjoys robustness to data noise and preserves freedom in the spare representation of data without constraints on the magnitude of the sparse codes.
The remaining parts of the paper are organized as follows. SRSC and its optimization algorithm, together with `2-RSC are introduced in the next section. The theoretical properties of the optimization of SRSC are shown in Section 3, with theoretical guarantee on the obtained sub-optimal solution for each step of the coordinate descent for obtaining the support regularized sparse codes: convergence to the critical point of the objective function and being close to the globally optimal solution. We then show the performance of the SRSC on data clustering, and conclude the paper. We use bold letters for matrices and vectors, and regular lower letter for scalars throughout this paper. The bold letter with superscript indicates the corresponding column of a matrix, and the bold letter with subscript indicates the corresponding element of a matrix or vector. ‖ · ‖F and ‖ · ‖p denote the Frobenius norm and the `p-norm.
2 SUPPORT REGULARIZED SPARSE CODING
2.1 CAPTURING LOCALLY LINEAR STRUCTURE: SUPPORT REGULARIZED SPARSE CODING
In this section, we introduce Support Regularized Sparse Coding (SRSC) which is designed to capture the locally linear structure of the data manifold for sparse coding. One of the most important properties of manifold is that it is locally Euclidean, and each data point in the manifold has a neighbourhood that is homeomorphic to a Euclidean space. The success of several manifold learning methods, including LLE (Roweis & Saul, 2000), SMCE (Elhamifar & Vidal, 2011) and Locally Linear Hashing (Irie et al., 2014), is built on exploiting the locally linear structure of manifold. In these methods, the locally linear structure associated with each data point is a linear representation of
that point by a set of its nearest neighbors in a nonparametric manner, from which the low-dimensional embedding complying to the manifold structure of the original data is obtained and used for various learning tasks. In the context of sparse coding, the data lie on or close to the subspaces spanned by the dictionary atoms specified by the nonzero elements of the corresponding sparse codes. Inspired by this observation, we propose to capture the locally linear structure of the data manifold for sparse coding by encouraging nearby data to share the atoms of the dictionary, so that nearby data are on or close to the local subspace spanned by the common dictionary atoms (see Figure 2).
In order to obtain the sparse codes with locally similar support so as to capture the locally linear structure of the data manifold, we propose Support Regularized Sparse Coding (SRSC), which uses support distance to measure the distance between the sparse codes of nearby data. Given a proper symmetric similarity matrix A, the sparse codes Z that capture the locally linear structure of the manifold minimizes the following support regularization term:
RA(Z) = 1
2 n∑ i=1 n∑ j=1 Aijd(Z i,Zj) (3)
A is usually the adjacency matrix of K-Nearest-Neighbor (KNN) graph, i.e. Aij = 1 if and only if xi is among the K nearest neighbors of xj or xj is among the K nearest neighbors of xi. Note that KNN is extensively used in the manifold learning literature, such as Locally Linear Embedding (LLE) (Roweis & Saul, 2000), Laplacian Eigenmaps (Belkin & Niyogi, 2003) and Sparse Manifold Clustering and Embedding (SMCE) (Elhamifar & Vidal, 2011), to establish the local neighborhood in the manifold. d indicates the support distance. For two vectors u,v of the same size, their support distance is defined below:
d(u,v) = |u|∑ t=1 (1Iut=0,vt 6=0 + 1Iut 6=0,vt=0) (4)
where 1I is the indicator function. When the support distance between Zi and Zj is small for nonzero Aij , xi and xj choose similar atoms of the dictionary for sparse representation. Therefore, SRSC captures the locally linear structure of the data manifold by encouraging nearby data to share dictionary atoms, wherein the common atoms shared by nearby data serve as the basis of the local subspace.
The optimization problem of SRSC is presented below:
min D,Z L(D,Z) = n∑ i=1 1 2 ‖xi −DZi‖22 + λ‖Zi‖1 + γRA(Z) s.t. ‖Dk‖2 ≤ 1, k = 1, . . . , p (5)
where γ > 0 is the weighting parameter for the support regularization term. Similar to (Lee et al., 2006), problem (5) is optimized alternatingly with respect to the dictionary D and the sparse codes Z respectively with the other variable fixed.
2.1.1 OPTIMIZING WITH RESPECT TO D WITH FIXED Z
The optimization with respect to D with fixed Z is a quadratic programming problem:
min D
1 2 ‖X −DZ‖2F s.t. ‖Dk‖2 ≤ 1, k = 1, . . . , p (6)
which can be solved using Lagrangian dual (Lee et al., 2006).
2.1.2 OPTIMIZING WITH RESPECT TO Z WITH FIXED D
We use coordinate descent to optimize (5) with respect to Z with fixed D:
min Z n∑ i=1 1 2 ‖xi −DZi‖22 + λ‖Zi‖1 + γRA(Z) (7)
In each step of coordinate descent, the optimization is performed over the i-th column of Z, while fixing all the other sparse codes {Zj}j 6=i. For each 1 ≤ i ≤ n, the optimization problem for Zi is below:
min Zi
F (Zi) = 1
2 ‖xi −DZi‖22 + λ‖Zi‖1 + γRA(Zi) (8)
where RA(Zi) = n∑ j=1 Aijd(Z i,Zj).
Inspired by recent advances in solving non-convex optimization problems by proximal linearized method (Bolte et al., 2014), proximal gradient descent method (PGD) is used to optimize the nonconvex problem (8). Although the proximal mapping is typically associated with a lower semicontinuous function (Bolte et al., 2014) and it can be verified that RA is not always lower semicontinuous, we can still derive a PGD-styple iterative method to optimize (8). Define GA ∈ IRp×n as GAki = n∑ j=1 Aij1IZkj=0 − n∑ j=1 Aij1IZkj 6=0 where 1I is the indicator function, then GAki indicates the degree to which Zki is discouraged to be nonzero and it can be verified that 1
RA(Z i) = p∑ k=1 GAki1IZki 6=0 (9)
Since each indicator function 1IZki 6=0 is lower semicontinuous, RA is lower semicontinuous if GAki ≥ 0 for k = 1, . . . , p. In the following text, we let Q(Zi) = 12‖xi −DZ
i‖22. The superscript with bracket indicates the iteration number of PGD or the iteration number of the coordinate descent without confusion. The PGD-style iterative method for optimizing (8) is as follows:
Z̃i (t) = Zi (t−1) − 1 τs (D>DZi (t−1) −D>xi) (10)
Zki (t) = { arg min v∈{uk,0} Hk(v) : uk 6= 0 or uk = 0 and GAki ≥ 0
ε : uk = 0 and G A ki < 0
(11)
for k = 1, . . . , p and ε is any real number such that ε 6= 0 and Hk(ε) ≤ Hk(Z(t−1)ki ). Hk and u are defined below:
Hk(v) = τs
2 (v − Z̃(t)ki ) 2 + λ|v|+ γGAki1Iv 6=0 (12)
for v ∈ IR and each 1 ≤ k ≤ p, and
u = max{|Z̃i (t) | − λ
τs , 0} ◦ sign(Z̃i
(t) ) (13)
where ◦ means element-wise multiplication. Proposition 1 shows that the PGD-style iterative method decreases the value of the objective function in each iteration.
1RA(Z i) is equal to the right hand side of (9) up to a constant.
Proposition 1. Let the sequence {Zi(t)}t be generated by the PGD-style iterative method with (10) and (11), then the sequence of the objective {F (Zi(t))}t decreases, and the following inequality holds for t ≥ 1:
F (Zi (t) ) ≤ F (Zi(t−1))− (τ − 1)s 2 ‖Zi(t) − Zi(t−1)‖22 (14)
And it follows that the sequence {F (Zi(t))}t converges. Remark 1. (10) and (11) in each iteration of the proposed PGD-style iterative method resemble that of the ordinary PGD. (10) performs gradient descent on the differential part, and (11) can be viewed as an approximate solution to the proximal mapping minv∈IRp H(v) = τs 2 ‖v − Z̃i (t) ‖22 + λ‖v‖1 + γRA(v). Since RA(Zi) is not always lower semicontinuous, arg minv∈IRp H(v) is not guaranteed to exist. One can see a simple example where this happens when uk = 0 and GAki < 0 for some k = 1, . . . , p, and in this case infv∈IRHk(v) = τs 2 (Z̃ (t) ki )
2 + γGAki but this infinitum can not be achieved.
In (10), τ > 1 is a constant and s is the Lipschitz constant for the gradient of function Q(·), namely ‖∇Q(y)−∇Q(z)‖2 ≤ s‖y − z‖2, ∀y, z ∈ IRp (15)
The PGD-style iterative method starts from t = 1 and continues until the sequence {F (Zi(t))} converges or maximum iteration number is achieved. When the iterative method converges or terminates for each Zi, the step of coordinate descent for Zi is finished and the optimization algorithm proceeds to optimize other sparse codes.
Algorithm 1 Support Regularized Sparse Coding
Input:
The data setX = {xi}ni=1, the parameter λ, γ, maximum iteration number M for the alternating method over D and Z, and maximum iteration number Mz for coordinate descent on Z, maximum iteration number Mp for the PGD-style iterative method on each Zi (i = 1,. . . ,n). and stopping threshold ε. 1: m = 1 2: while m ≤M do 3: Perform coordinate descent to optimize (7) and obtain Z(m) with fixed D(m−1). In i-th (1 ≤ i ≤ n) step
of each iteration of coordinate descent, solve (8) using the PGD-style iterative method (10) and (11) to update Zi in each iteration of the PGD-style iterative method.
4: Optimize (6) using Lagrangian dual and obtain D(m) with fixed Z(m). 5: if |L(D(m),Z(m))− L(D(m−1),Z(m−1))| < ε then 6: break 7: else 8: m = m+ 1. 9: end if
10: end while Output: the support regularized sparse codes Ẑ when the above iterations converge or maximum iteration
number is achieved.
TIME COMPLEXITY
Algorithm 1 describes the algorithm of SRSC. We solve the ordinary sparse coding problem (1) by the online dictionary learning method (Mairal et al., 2009) and use the dictionary and the sparse codes as the initialization D(0) and Z(0) for the alternating method in Algorithm 1. In Algorithm 1, the time complexity of optimization over the sparse codes is O(MMzMpndp2), and time complexity of optimization over the dictionary using Newton’s method to solve the Lagrangian dual problem is O ( M ( np2 + Tnewton(3p 2.807 + 2dp2 + dnp) )) , where Tnewton is the maximum iteration number
for Newton’s method. Therefore, the overall time complexity of Algorithm 1 isO ( M ( MzMpndp 2 +
np2 + Tnewton(3p 2.807 + 2dp2 + dnp) )) . It should be emphasized that the optimization over the
dictionary for SRSC has the same efficiency as the efficient sparse coding method (Lee et al., 2006),
and the optimization over the sparse code of each data point by the PGD-style iterative method (10) and (11) is almost as efficient as the widely used Iterative Shrinkage and Thresholding Algorithm (ISTA) (Daubechies et al., 2004; Beck & Teboulle, 2009). Note that step (10) and (13) are required by both our method and ISTA; compared to ISTA, the extra operations incurred by our PGD-style iterative method (10) and (11) are only the arithmetic operations with time complexity 20p for evaluating the function Hk(·) defined in (12) for k = 1, . . . , p. More specifically, evaluating the value of function Hk(v) takes 10 arithmetic operations and two evaluations at v = uk and v = 0 are needed. Since a compact dictionary is preferred by the extensive study of the sparse coding and dictionary learning literature and the dictionary size p ≤ 500 is adopted throughout our experiments, our PGD-style iterative method only incurs extra operations of constant time complexity compared to ISTA while learning supported regularized sparse codes. In Section 4, we propose Deep-SRSC as a fast approximation of SRSC with considerable speedup for obtaining the sparse codes of the new data or the test data (see more details in Section 4.1). Furthermore, we conduct the empirical study and show that the parallel coordinate descent method, which updates the codes of a group of P data points in parallel and provides P times speedup over the coordinate descent method used in Section 2.1.2 and Algorithm 1, exhibits almost the same performance as the coordinate descent method for the clustering task on the test set of the CIFAR-10 data. Please refer to the details in the subsection “Deep-SRSC with the Second Test Setting (Referring to the Training Data)” in the Appendix.
2.2 RELATED WORK: `2 REGULARIZED SPARSE CODING (`2-RSC)
The manifold assumption (Belkin et al., 2006) is usually employed by existing regularized sparse coding methods (Liu et al., 2010; He et al., 2011; Zheng et al., 2011; Gao et al., 2013) to obtain the sparse code according to the manifold structure of the data. Interpreting the sparse code of a data point as its embedding, the manifold assumption in the case of sparse coding for most existing methods requires that if two points xi and xj are close in the intrinsic geometry of the submanifold, their corresponding sparse codes Zi and Zj are also expected to be similar to each other in the sense of `2-distance (Zheng et al., 2011; Gao et al., 2013). In other words, z varies smoothly along the geodesics in the intrinsic geometry. Based on the spectral graph theory (Chung, 1997), extensive literature uses graph Laplacian to impose local smoothness of the embedding and preserve the local manifold structure (Belkin et al., 2006; Zheng et al., 2011; Gao et al., 2013).
The sparse code Z that captures the local geometric structure of the data in accordance with the manifold assumption by graph Laplacian minimizes the following `2 regularization term, or the Laplacian regularization term:
R (`2) A (Z) = 1
2 n∑ i=1 n∑ j=1 Aij‖Zi − Zj‖22 (16)
where the `2-norm is used to measure the distance between sparse codes, and A is the same as that in Section 2.1. LA = DA −A is the graph Laplacian associated with the similarity matrix A, the degree matrix DA is a diagonal matrix with each diagonal element being the sum of the
elements in the corresponding row of A, namely (DA)ii = n∑ j=1 Aij . To the best of our knowledge, such `2 regularization is employed by most methods that use graph regularization for sparse coding. Incorporating the `2 regularization term into the optimization problem of sparse coding (1), the formulation of `2 Regularized Sparse Coding (`2-RSC) is presented below
min D,Z
L(` 2)(Z) = n∑ i=1 ‖xi −DZi‖22 + λ‖Zi‖1 + γ(` 2)R (`2) A (Z) s.t. ‖D k‖2 ≤ 1, k = 1, . . . , p (17)
ADVANTAGE OF SRSC OVER `2-RSC
Although `2-RSC imposes the local smoothness on the sparse codes, it does not capture the locally linear structure of the data manifold. By promoting the smoothness on the support of the sparse codes rather than their `2-distance, SRSC encodes the locally linear structure of the manifold in the sparse codes while reserving freedom in the sparse representation of the data with no constraints on the
magnitude of the sparse codes. Moreover, as pointed out by (Wang et al., 2015), support regularization offers robustness to noise for sparse coding. In SRSC, all the data consult their neighbors for choosing the dictionary atoms rather than choosing the atoms on their own, and the sparse codes of the noisy data are suppressed since they are forced to choose similar or the same atoms as the nearby clean data instead of choosing the atoms in the interests of representing themselves.
3 THEORETICAL ANALYSIS
It can be observed that optimization by coordinate descent over the sparse code in Section 2.1.2 is important for the overall optimization of SRSC, and each step of the coordinate descent (8) is a difficult nonconvex problem and crucial for obtaining the support regularized sparse code, where the nonconvexity comes from the support regularization term RA(Zi) (9). Therefore, the optimization of (8) plays an important role in the overall optimization of SRSC. In the previous section, a PGDstyle iterative method is proposed to decrease the value of the objective in each iteration. In this section, we provide further theoretical analysis on the optimization of problem (8) when GAki ≥ 0 for k = 1, . . . , p. This condition is equivalent to the condition that the support regularization function
Rc(v) , p∑ k=1 ck1Ivk 6=0 (18)
is lower semicontinuous, where c ∈ IRp is the coefficients and ck = GAki. Under this condition, we prove that the sequence {Zi(t)}t produced by the PGD-style iterative method converges to the suboptimal solution which is a critical point of the objective (8). By connecting the support regularized function to the capped-`1 norm and the nonconvexity analysis of the support regularization term, we present the bound for `2-distance between the sub-optimal solution and the globally optimal solution to (8) in Theorem 1. Note that our analysis is valid for all 1 ≤ i ≤ n. We first have the following result that the support regularization function (18) is lower semicontinuous if and only if all the coefficients c are nonnegative. Proposition 2. The support regularization function (18) is lower semicontinuous if and only if all the coefficients c are nonnegative.
Therefore, if GAki ≥ 0 for k = 1, . . . , p, the support regularization term RA(Zi) is lower semicontinuous with respect to Zi in (9). In this case, the PGD-style iterative method proposed in Section 2.1.2 for each iteration t ≥ 1 becomes
Z̃i (t) = Zi (t−1) − 1 τs (D>DZi (t−1) −D>xi) (19)
Zki (t) = arg min
v∈{uk,0} Hk(v), k = 1, . . . , p (20)
which is equivalent to the updates rules in the ordinary proximal gradient descent method. It is worthwhile to mention the meaning of the condition that GAki ≥ 0 for k = 1, . . . , p. For a data point xi, if the number of its neighbors with zero k-th element of the sparse codes is larger than that with nonzero k-th element of the sparse codes, which indicates that the neighbors of xi suggest that a zero k-th element of the sparse code of xi is preferable, then GAki ≥ 0 and GAki quantitatively represents the penalty if the sparse code element Zik is nonzero while the neighbors of xi suggest that Z i k = 0 is preferable. Intuitively, this situation happens when there is conflict between choosing the support of the code solely by the data point itself and the suggestion of its neighbors; if the point is an outlier or suffering from noise, the optimization can help that point make a sensible choice by considering the suggestion of its neighbors. We observe that GAki ≥ 0 for k = 1, . . . , p happens in all the data sets used in this paper.
In the following lemma, we show that the sequence {Zi(t)}t generated by (19) and (20) converges to a critical point of F (Zi), denoted by Ẑi. Denote by Zi∗ the globally optimal solution to the original optimization problem (8). The following lemma also shows that both Ẑi and Zi∗ are local solutions to the capped-`1 regularized problem (21). Before stating the lemma, the following definitions are introduced which are essential for our analysis.
Definition 1. (Critical points) Given the non-convex function f : IRn → R ∪ {+∞} which is a proper and lower semi-continuous function.
• for a given x ∈ domf , its Frechet subdifferential of f at x, denoted by ∂̃f(x), is the set of all vectors u ∈ IRn which satisfy
lim sup y 6=x,y→x f(y)− f(x)− 〈u,y − x〉 ‖y − x‖ ≥ 0
• The limiting-subdifferential of f at x ∈ IRn, denoted by written ∂f(x), is defined by
∂f(x) = {u ∈ IRn : ∃xk → x, f(xk)→ f(x), ũk ∈ ∂̃f(xk)→ u}
The point x is a critical point of f if 0 ∈ ∂f(x).
Also, we are considering the following capped-`1 regularized problem, which replaces the indicator function in the support regularization term RA(Zi) with the continuous capped-`1 regularization term T:
min β∈IRp
Lcapped−`1(β) = 1
2 ‖xi −Dβ‖22 + λ‖β‖1 + T(β; b) (21)
where T(β; b) = p∑ k=1 Tk(βk; b), Tk(t; b) = γGAki min{|t|,b} b for some b > 0. It can be seen that the
objective function of the capped-`1 problem approaches that of (8) when min{|t|,b}b approaches the indicator function 1It6=0 as b→ 0+. Define P(·; b) = λ‖ · ‖1 + T(·; b), the location solution to the capped-`1 problem is defined as follows.
Definition 2. (Local solution) A vector β̃ is a local solution to the problem (21) if
‖D>(Dβ̃ − xi) + Ṗ(β̃; b)‖2 = 0 (22)
where Ṗ(β̃; b) = [Ṗ1(β̃1; b), Ṗ2(β̃2; b), . . . , Ṗp(β̃p; b)]>, Pk(t; b) = λ|t| + Tk(t; b) for k = 1, . . . , p.
Note that in the above definition and the following text, Ṗk(t; b) can be chosen as any value between the right differential ∂Pk∂t (t+; b) (or Ṗk(t+; b)) and left differential ∂Pk ∂t (t−; b) (or Ṗk(t−; b)) for k = 1, . . . , p.
Definition 3. (Degree of Nonconvexity of a Regularizer) For κ ≥ 0 and t ∈ IR, define
θ(t, κ) := sup s {−sgn(s− t)(Ṗ (s; b)− Ṗ (t; b))− κ|s− t|}
as the degree of nonconvexity for function P . If u = (u1, . . . , up)> ∈ IRp, θ(u, κ) = [θ(u1, κ), . . . , θ(up, κ)]. sgn is the sign function.
Note that θ(t, κ) = 0 for convex function P .
Let Ŝi = supp(Ẑi) where supp(·) indicates the support of a vector, i.e. the indices of its nonzero elements. Denote by Zi∗ the globally optimal solution to (8), and S∗i = supp(Z i∗), then we have
Lemma 1. For any 1 ≤ i ≤ n, if GAki ≥ 0 for k = 1, . . . , p, then the sequence {Zi (t)}t generated by (10) and (11) converges to a critical point of F (Zi), which is denoted by Ẑi. Moreover, if
0 < b < min{min j∈Ŝi |Ẑij |, max k/∈Ŝi,GAki 6=0 γGAki ( ∂Q ∂Zi
k
|Zi=Ẑi − λ)+ , min j∈S∗i
|Zij ∗|, max
k/∈S∗i ,G A ki 6=0 γGAki ( ∂Q ∂Zij |Zi=Zi∗ − λ)+ }
(23)
(if the denominator is 0, ·0 is defined to be +∞ in the above inequality), then both Ẑ i and Zi∗ are
local solutions to the capped-`1 regularized problem (21).
Using the degree of nonconvexity of the regularizer P, we have the following theorem showing that the sub-optimal solution Ẑi obtained by our PGD-style iterative method can be close to the globally optimal solution to the original problem (8), i.e. Zi∗. In the following text, BI indicates a submatrix of B whose columns correspond to the nonzero elements of I, and σmin(·) indicates the smallest singular value of a matrix. Theorem 1. (Sub-optimal solution is close to the globally optimal solution) For any 1 ≤ i ≤ n, let Ei = Ŝi ∪ S∗i . Suppose GAki ≥ 0 for k = 1, . . . , p, DEi is not singular with κ0 , σmin(DEi) > 0, κ20 > κ > 0, and b is chosen according to (23) as in Lemma 1. Let S̃i = (Ŝi \ S∗i )∪ (S∗i \ Ŝi) be the symmetric difference between Ŝi and S∗i , then
‖Zi∗ − Ẑi‖2 ≤ 1
κ20 − κ (( ∑ k∈S̃i∩Ŝi (max{0, γG A ki b − κ|Ẑki − b|})2 + ∑ k∈S̃i\Ŝi (max{0, γG A ki b − κb})2 ) 1 2 + ‖t‖2 ) (24)
where t ∈ IRp, tk = 2λ1IZik∗Ẑik<0 + 01IZik∗Ẑik>0 for k ∈ Ŝi ∩ S ∗ i , and tk = 0 for all other k.
Remark 2. Note that the bound for distance between the sub-optimal solution and the globally optimal solution presented in Theorem 1 does not require typical Restricted Isometry Property (RIP) conditions, e.g. Cands (2008). Also, when γG A ki
b − κ|Ẑki − b| and γGAki b − κb are no greater than
0 and Zi∗ and Ẑi has the same sign in the intersection of their support, the sub-optimal solution Ẑi is equal to the globally optimal solution. When γG A ki
b − κ|Ẑki − b| and γGAki b − κb are small
positive numbers and Zi∗ and Ẑi has similar sign in the intersection of their support, Ẑi is close to the globally optimal solution.
4 DEEP SUPPORT REGULARIZED SPARSE CODING: DEEP-SRSC
Inspired by the PGD-style iterative method (10) and (11) for SRSC and the LISTA network, we propose Deep Support Regularized Sparse Coding (Deep-SRSC) illustrated in Figure 3, which is a neural network that produces the approximate support regularized sparse codes for SRSC. The goal of Deep-SRSC is to approximate the sparse codes of the input data in a fast way by feeding the data through the Deep-SRSC network, instead of running the iterative optimization algorithm for SRSC in Section 2.1. To achieve this goal, the Deep-SRSC network is trained on the training data by minimizing the squared distance between the predicted codes of the training data by the network and their ground truth codes. The network design of Deep-SRSC is in accordance with the proposed PGD-style iterative method. When W = 1LD >, S = I− 1LD >D where L = τs, then each stage in the recurrent structure of Deep-SRSC implements one iteration of PGD-style iterative method, i.e. (10) and (11). In Deep-SRSC, W, S and L are to be learned by the network rather than computed from a pre-computed dictionary D, and S is shared over different layers. The min-pooling neuron in Deep-SRSC outputs the result of arg min
v∈{uk,0} Hk(v) or ε, according to the update rule (11). Figure 3
illustrates Deep-SRSC with 2 layers.
Denote the training data by x1, . . . ,xm, and let Zsr be the ground truth support regularized sparse codes of the training data which are obtained by the optimization method introduced in Section 2.1. Let fsr be the Deep-SRSC encoder which produces the approximate support regularized sparse code z = fsr(x,Θsr), where Θsr = (W,S, L) denotes the parameters of Deep-SRSC. Then the parameters of Deep-SRSC are learned by minimizing the cost function which measures the distance between the predicted approximate support regularized sparse codes and the ground truth ones: 1 m m∑ i=1 ‖Zisr − fsr(xi,Θsr)‖22. Similar to the LISTA network, the optimization is performed by
stochastic gradient descent and back-propagation. The batch size is set to 1 so as to simulate the coordinate descent method for optimization over the sparse codes in Section 2.1.2. The adjacency matrix of the KNN graph over the training data is required as input for training the network.
The approximate codes of the new data, or the test data, are obtained by feeding the new data through the Deep-SRSC network learned on the training data. We provide two test settings below, depending on whether training data are referred to in the test process.
1) In the first setting where the training data are not referred to, the test data are a group of data points. The test data and the KNN graph over them are fed into the Deep-SRSC network to obtain the approximate codes of the test data. The locally linear manifold structure of the test data is encoded in the KNN graph over the test data. This setting is potentially more suitable for the situation of limited storage where the training data and their codes do not need to be stored in the test process. This setting may not be suitable for the test data that do not reliably reflect the locally linear manifold structure (e.g. in the case of a very small amount of test data), and in this case the second setting below is a better choice.
2) In the second setting where training data are referred to, the approximate code of each data point is obtained by feeding that point and the KNN graph over that point and the training data into the Deep-SRSC network. The code of each test point is reliably obtained by referring to its nearest neighbors in the training data and this process is independent of the factor that whether the test data reflect the locally linear manifold structure.
4.1 DEEP-SRSC AS FAST ENCODER
It should be emphasized that Deep-SRSC is a fast encoder for SRSC when obtaining the codes of the new data (or test data). Each layer of Deep-SRSC resembles one iteration of the PGD-style iterative method (10) and (11), and the computational cost of feeding forward a data point through one layer is the same as that of executing one iteration of the PGD-style iterative method for that point. Therefore, the feed-forward process of obtaining the sparse codes of the new data using `-layer Deep-SRSC is around Mp` times faster than the PGD-style iterative method used in Algorithm 1, where Mp is the maximum iteration number for the PGD-style iterative method. In the experimental results shown in the next section, Deep-SRSC with different number of layers are employed to produce the approximate support regularized sparse codes, and 6-layer Deep-SRSC achieves minimum prediction error. With Mp = 50 throughout our experiments, Deep-SRSC is around 506 ≈ 8.3 times faster than the PGD-style iterative method. Our analysis in this subsection holds for both test settings.
5 EXPERIMENTAL RESULTS
5.1 CLUSTERING PERFORMANCE
In this subsection, the superiority of SRSC is demonstrated by its performance in data clustering on various data sets, e.g. USPS handwritten digits data set, COIL-20, COIL-100 and UCI Gesture Phase Segmentation data set. Two measures are used to evaluate the performance of the clustering methods, i.e. the Accuracy (AC) and the Normalized Mutual Information (NMI) (Zheng et al., 2004). SRSC is compared to K-means (KM), Spectral Clustering (SC), Sparse Coding and `2-RSC in Section 2.2. Throughout all the experiments, we set K = 3 for building the adjacency matrix A of KNN graph, dictionary size p = 300 and λ = 0.1 for both `2-RSC and SRSC. We also set γ(`
2) = 1 which is the suggested default value in (Zheng et al., 2011), and M = Mz = 5 and Mp = 50 in Algorithm 1. The default value of the weight for support regularization term of SRSC is γ = 0.5. SRSC is implemented by both MATLAB and CUDA C++ with extreme efficiency, and the code is published on GitHub: https://github.com/yingzhenyang/SRSC.
The USPS handwritten digits data set is comprised of n = 9298 handwritten images of ten digits from 0 to 9, and each image is of size 16 × 16 and represented by a 256-dimensional vector. The whole data set is divided into training set of 7291 images and test set of 2007 images. We run Algorithm 1 to obtain the support regularized sparse code Ẑ, then build a n× n similarity matrix Y over all the data. Two similarity measure are employed: the first similarity is the positive part of the inner product of their corresponding sparse codes, namely Yij = max{0, Ẑi > Ẑj}, the second one is Yij = Aijq > Ẑi q Ẑj
where qv is a binary vector of the same size as v with element 1 at the indices of nonzero elements of v. The second similarity measure is name the support similarity and it considers the number of common dictionary atoms chosen by the sparse codes. Spectral clustering is performed on the similarity matrix Y to obtain the clustering result of SRSC, and the best performance among the two similarity measures is reported. The same procedure is performed by all the other sparse coding based methods to obtain clustering results. The clustering results of various methods are shown in Table 1.
COIL-20 Database has 1440 images of resolution 32 × 32 for 20 objects, and the background is removed in all images. The dimension of this data is 1024. Its enlarged version, COIL-100 Database, contains 100 objects with 72 images of resolution 32 × 32 for each object. The images of each object were taken 5 degrees apart when each object was rotated on a turntable. The UCI Gesture Phase Segmentation data set contains the gesture information of three users when they told stories of some comic strips in front of the Microsoft Kinect sensor. We use the processed file provided by the original data consisting of 9873 frames, and the gesture information in each frame is the vectorial velocity and acceleration of left hand, right hand, left wrist, and right wrist, represented by a 32-dimensional vector. The clustering results on these three data sets are shown in Table 2. It can be observed from Table 1 and Table 2 that SRSC always produces better clustering accuracy than other competing methods, due to its capability of capturing the locally linear manifold structure of the data and robustness to noies. In the appendix, we further show the performance of different sparse coding based methods with different dictionary size on COIL-100 data set in Table 5, and investigate the parameter sensitive of SRSC by demonstrating its performance with varying γ and K in Table 6.
5.2 APPROXIMATION BY DEEP-SRSC
In this subsection, Deep-SRSC is employed as a fast encoder to approximate the support regularized sparse codes of SRSC on the USPS data set. Throughout this subsection, we show results using the first test setting introduced in Section 4, i.e. test without referring to the training data. Additional experimental results on the performance of Deep-SRSC with the second test setting, including the application to semi-supervised learning by label propagation (Zhu et al., 2003), are shown in the appendix.
The Deep-SRSC network is trained on the training set of the USPS data comprising 7291 images. We adopt three depth settings wherein Deep-SRSC has 1 layer, 2 layers, and 6 layers respectively. We first run SRSC on the training set of USPS data to obtain the dictionary Dsr and the support regularized sparse codes Zsr of the training data. Then the optimization problem (7) is solved by the PGD-style iterative method in Section 2.1.2, whereX is the test data, A is the adjacency matrix of the KNN graph over the test data, to obtain the support regularized sparse codes Zsr,test of the test data with dictionary Dsr. Zsr is used as the ground truth support regularized sparse codes to train Deep-SRSC, and Zsr,test serves as the ground truth codes of the test data. The approximate codes of the test data of the USPS data are obtained by feeding forward them into the Deep-SRSC network together with the KNN graph over the test data, and the prediction error of Deep-SRSC is the average of the squared error between the predicted codes and Zsr,test. Figure 4 illustrates the training error of Deep-SRSC w.r.t. the epoch number for 1 layer, 2 layers, and 6 layers respectively, and Figure 5 in the appendix illustrates the test error of Deep-SRSC. For each depth setting, Deep-SRSC is trained with 300 epoches, and testing is performed for every 5 epoches during training. It can be observed that deeper Deep-SRSC leads to smaller training and test error. Deep-SRSC is implemented with TensorFlow (Abadi et al., 2016). The initial learning rate is set to 10−4, and divided by 10 at 100-th epoch and 200-th epoch, so the final learning rate is 10−6 upon the termination of the training.
Table 3 shows the prediction error of Deep-SRSC for different dictionary size p and different number of layers. It can be observed that Deep-SRSC with more layers demonstrates smaller prediction error for the same dictionary size due to its better representation capability, and smaller dictionary size leads to less prediction error for the same number of layers due to the reduced difficulty of representation. Moreover, the codes predicted by 6-layer Deep-SRSC are used to perform clustering on the test data because of its minimum prediction error, with comparison to the performance of sparse coding and `2-RSC shown in Table 4 with respect to different dictionary size. For either sparse coding or `2-RSC, the dictionary is firstly learned on the training data, then the sparse codes of the test data obtained with respect to that dictionary are used to perform clustering on the test set of USPS data. We can see that SRSC together with its approximation, 6-layer Deep-SRSC, achieve the highest accuracy and NMI. In addition, a reasonably large dictionary benefits SRSC, e.g. increasing p from 100 to 300 boosts its accuracy, since the dictionary atoms serve as the basis for the locally linear structures (local subspaces) of the data manifold and a sufficiently large dictionary size is favorable for modeling all such locally linear structures. On the other hand, a too large dictionary (such as p = 500) imposes much difficulty on the optimization which can even hurt the performance of SRSC, `2-RSC and regular sparse coding.
6 CONCLUSION
We propose Support Regularized Sparse Coding (SRSC) which captures the locally linear manifold structure of the high-dimensional data for sparse coding and enjoys robustness to noise. SRSC achieves this goal by encouraging nearby data in the manifold to share dictionary atoms. The optimization algorithm of SRSC is presented with theoretical guarantee for the optimization over the sparse codes. In addition, we propose Deep-SRSC, a feed-forward neural network, as a fast encoder to approximate the support regularized sparse codes produce by SRSC. Experimental results demonstrate the effectiveness of SRSC by its application to data clustering, and show that Deep-SRSC renders approximate codes for SRSC with low prediction error. The approximate codes generated by 6-layer Deep-SRSC also deliver compelling empirical performance for data clustering.
APPENDIX
PROOFS
Proof of Proposition 1. Note that u is the optimal solution to the lasso problem arg minv∈Rp τs 2 ‖v− Z̃(t)ki ‖ 2 2 + λ‖v‖1. Define Tk(v) = τs2 (v − Z̃ (t) ki ) 2 + λ|v| for v ∈ IR, then uk = arg minv∈IR Tk(v). Since the
two functions Hk(v) and Tk(v) only differ at v = 0, arg minv∈{uk,0}Hk(v) is the optimal solution to minv∈RHk(v) when uk 6= 0 or uk = 0 and GAki ≥ 0.
When uk = 0 and GAki < 0, when ε→ 0 and ε 6= 0, Hk(v)→ GAki and infv∈IRHk(v) = τs2 (Z̃ (t) ki ) 2 + γGAki. Note that the infimum can never be achieved. Since infv∈IRHk(v) < Hk(Z (t−1) ki ), we can always find ε 6= 0 such that Hk(ε) ≤ Hk(Z(t−1)ki ).
Define H(v) = τs 2 ‖v − Z̃i (t) ‖22 + λ‖v‖1 + γRA(v). Based on the above argument, H(Zi (t) ) ≤ H(Zi (t−1) ) which indicates that
τs 2 ‖Zi(t) − Zi(t−1)‖22 + 〈Zi (t) − Zi(t−1),∇Q(Zi(t−1))〉 (25)
+ λ‖Zi(t)‖1 + γRA(Zi (t) ) ≤ λ‖Zi(t−1)‖1 + γRA(Zi (t−1) ) (26)
Also, since s is the Lipschitz constant for the gradient of function Q(·), we have
Q(Zi (t) ) ≤ Q(Zi(t−1)) + 〈Zi(t) − Zi(t−1),∇Q(Zi(t−1))〉+ s 2 ‖Zi(t) − Zi(t−1)‖22 (27)
Combining (25) and (27),
F (Zi (t) ) ≤ F (Zi(t−1))− (τ − 1)s 2 ‖Zi(t) − Zi(t−1)‖22
Proof of Lemma 1. We first prove that the sequences {Zi(t)}t is bounded for any 1 ≤ i ≤ n. By Proposition 1, the sequence {F (Zi(t))}t decreases, so we have
F (Zi (t) ) = 1
2 ‖xi −DZi (t)‖22 + λ‖Zi (t)‖1 + γRA(Zi (t) )
≤ ‖xi −DZi (0)‖22 + λ‖Zi (0)‖0 ≤ 1 + RA(Zi (0) )
for t ≥ 1. Therefore,
‖Zi(t)‖1 ≤ 1
λ ‖xi −DZi (0)‖22 + λ‖Zi (0)‖0 ≤ 1 + RA(Zi (0) )
It follows that ‖Zi(t)‖1 is bounded, and ‖Zi (t)‖2 is also bounded. Since GAki ≥ 0 for k = 1, . . . , p and
the indicator function 1I·6=0 is semi-algebraic function, RA(·) is also a semi-algebraic function and lower semicontinuous. Therefore, according to Theorem 1 by Bolte et al. (2014), {Zi(t)}t converges to a critical point of F (Zi), denoted by Ẑi.
Let v̂ = D>(DẐi − xi) + Ṗ(Ẑi; b). For k such that GAki = 0, since Ẑi is a critical point of F (Zi), v̂k = 0.
Now we consider the case that GAki 6= 0.
For for k ∈ Ŝi, since Ẑi is a critical point of F (Zi) = 12‖xi − DZ i‖22 + λ‖Zi‖1 + γRA(Zi). then ∂(Q+λ‖Zi‖1) ∂Zi
k
| Zi=Ẑi
= 0 because ∂RA(Z i)
∂Zi k
| Zi=Ẑi = 0 . Note that mink∈Ŝi |Ẑ i k| > b, so ∂T ∂Zi
k
| Zi=Ẑi = 0, and
it follows that v̂k = 0.
For k /∈ Ŝi, since dPkdZi k
(Ẑik+; b) = γGAki b +λ and dPk dZi
k
(Ẑik−; b) = − γGAki b −λ, γG A ki b +λ ≥ | ∂Q ∂Zi
k
| Zi=Ẑi |, we
can choose the k-th element of Ṗ(Ẑi; b) such that v̂k = 0. Therefore, ‖v̂‖2 = 0, and Ẑi is a local solution to the problem (21).
Now we prove that Zi∗ is also a local solution to (21). Let v∗ = D>(DZi∗ − xi) + Ṗ(Zi ∗ ; b), and Q is defined as before. For k such that GAki = 0, since Z i∗ is the globally optimal solution of F (Zi), v∗k = 0.
Again we consider the case that GAki 6= 0.
For k ∈ S∗i , since Zi ∗ is the globally optimal solution to problem (8), we also have ∂(Q+λ‖Z i‖1) ∂Zi
k
|Zi=Zi∗ = 0.
If it is not the case and ∂(Q+λ‖Z i‖1)
∂Zi k
|Zi=Zi∗ 6= 0, then we can change Zik by a small amount in the direction
of the gradient ∂(Q+λ‖Z i‖1)
∂Zi k
at the point Zi = Zi∗ while Zik is still nonzero, leading to a smaller value of the
objective F (Zi).
Note that mink∈S∗i |Z i k ∗| > b, so ∂T
∂Zi k
| Zi=Ẑi = 0, and it follows that v∗k = 0.
For k /∈ S∗i , since γGAki b + λ ≥ maxk/∈Ŝi | ∂Q
∂Zi k
|Zi=Zi∗ |, we can choose the k-th element of Ṗ(Zi ∗ ; b) such
that v∗k = 0. It follows that ‖v∗‖2 = 0, and Zi ∗ is also a local solution to the problem (21).
Proof of Theorem 1. According to Lemma 1, both Ẑi and Zi∗ are local solutions to problem (21). In the following text, let βI indicates a vector whose elements are those of β with indices in I. Let ∆ = Zi
∗ − Ẑi, ∆̃ = Ṗ(Zi ∗ )− Ṗ(Ẑi). By Lemma 1, we have
‖D>D∆ + ∆̃‖2 = 0 It follows that
∆>D>D∆ + ∆>∆̃ ≤ ‖∆‖2‖D>D∆ + ∆̃‖2 = 0
Also, by the proof of Lemma 1, for k ∈ Ŝi ∩ S∗i , since (D>D∆)k = 2λ1IZi k ∗Ẑi k <0 + 01IZi k ∗Ẑi k >0 we have
∆̃k = −(D>D∆)k. We now present another property on any nonconvex function P using the degree of nonconvexity in Definition 3: θ(t, κ) := sups{−sgn(s− t)(Ṗ (s; b)− Ṗ (t; b))− κ|s− t|} on the regularizer P. For any s, t ∈ IR, we have
− sgn(s− t) ( Ṗ (s; b)− Ṗ (t; b) ) − κ|s− t| ≤ θ(t, κ)
by the definition of θ. It follows that θ(t, κ)|s− t| ≥ −(s− t) ( Ṗ (s; b)− Ṗ (t; b) ) − κ(s− t)2
− (s− t) ( Ṗ (s; b)− Ṗ (t; b) ) ≤ θ(t, κ)|s− t|+ κ(s− t)2 (28)
Applying (28) with P = Pk for k = 1, . . . , p, we have
∆>D>D∆ ≤ −∆>∆̃ = −∆>S̃i∆̃S̃i −∆ > Ŝi∩S∗i ∆̃Ŝi∩S∗i
≤ |ZiS̃i ∗ − ẐiS̃i | >θ(ẐiS̃i , κ) + κ‖Z i S̃i ∗ − ẐiS̃i‖ 2 2 + ‖∆Ŝi∩S∗i ‖2‖∆̃Ŝi∩S∗i ‖2 ≤ ‖θ(ẐiS̃i , κ)‖2‖Z i S̃i ∗ − ẐiS̃i‖2 + κ‖Z i S̃ ∗ −Ẑ i S̃‖ 2 2 + ‖∆‖2‖∆̃Ŝi∩S∗i ‖2 ≤ ‖θ(ẐiS̃i , κ)‖2‖∆‖2 + κ‖∆‖ 2 2 + ‖∆‖2‖∆̃Ŝi∩S∗i ‖2 (29)
On the other hand, ∆>D>D∆ ≥ κ20‖∆‖22. It follows from (29) that
κ20‖∆‖22 ≤ ‖θ(ẐiS̃i , κ)‖2‖∆‖2 + κ‖∆‖ 2 2 + ‖∆‖2‖∆̃Ŝi∩S∗i ‖2
When ‖∆‖2 6= 0, we have
κ20‖∆‖2 ≤ ‖θ(ẐiS̃i , κ)‖2 + κ‖∆‖2 + ‖∆̃Ŝi∩S∗i ‖2
⇒ ‖∆‖2 ≤ ‖θ(Ẑi S̃i , κ)‖2 + ‖∆̃Ŝi∩S∗i ‖2 κ20 − κ
(30)
According to the definition of θ, it can be verified that θ(t, κ) = max{0, γG A ki b − κ|t − b|} for |t| > b, and θ(0+, κ) = max{0, γG A ki b − κb}. Therefore,
‖θ(ẐiS̃i , κ)‖2 = ( ∑ k∈S̃i∩Ŝi (max{0, γG A ki b − κ|Ẑki − b|})2+
∑ k∈S̃i\Ŝi (max{0, γG A ki b − κb})2 ) 1 2 (31)
Therefore,
‖∆‖2 ≤ 1
κ20 − κ (( ∑ k∈S̃i∩Ŝi (max{0, γG A ki b − κ|Ẑki − b|})2+
∑ k∈S̃i\Ŝi (max{0, γG A ki b − κb})2 ) 1 2 + ‖∆̃Ŝi∩S∗i ‖2 ) (32)
where ∆̃k = −(D>D∆)k = −2λ1IZi k ∗Ẑi k <0 − 01IZi k ∗Ẑi k >0 for k ∈ Ŝi ∩ S ∗ i . This proves the result of this
theorem.
MORE EXPERIMENTAL RESULTS
The test error of Deep-SRSC with different dictionary size, corresponding to Figure 4 showing the training error, is illustrated in Figure 5. We vary the dictionary size and show the clustering results on COIL-100 data set in Table 5, and we can see that SRSC always achieves highest accuracy and NMI with different dictionary size.
In addition, we investigate the parameter sensitivity of SRSC, and show in Table 6 the performance change while varying γ, the weight for the support regularization term, and K, the number of nearest neighbors when building the KNN graph for the support regularization term, on the USPS data set. It can be observed that the performance of SRSC is stable over a relatively large range of λ and K. SRSC often has the highest NMI while maintaining a very competitive accuracy.
DEEP-SRSC WITH THE SECOND TEST SETTING (REFERRING TO THE TRAINING DATA)
We demonstrate the performance of SRSC and Deep-SRSC with the second test setting (referring to the training data) on clustering and semi-supervised learning. The ground truth code of the each test data point is computed by performing the PGD-style iterative method to solve the problem (8) where xi is the test point, D is Dsr obtained from the training data as in Section 5.2, A is the adjacency matrix of the KNN graph over the test point and the training data. Table 9 shows the prediction error of Deep-SRSC for different dictionary size p and different number of layers on the USPS data, which is comparable to the case of the first test setting in Table 3.
Two more data sets are used in this subsection, i.e. MNIST for hand-written digit recognition and CIFAR-10 for image recognition. MNIST is comprised of 60000 training images and 10000 test images of ten digits from 0 to 9, and each image is of size 28× 28 and represented as a 784-dimensional vector. CIFAR-10 consists of 50000 training images and 10000 testing images in 10 classes, and each image is a color one of size 32× 32. Using the second test setting, Deep-SRSC is trained on the training set, and the codes of the test set predicted by 6-layer Deep-SRSC are used to perform clustering on the test set for MNIST and CIFAR-10 data, with comparison to other sparse coding based methods. The clustering results are shown in Table 7 and 8 respectively with dictionary size p = 300. We observe that SRSC and Deep-SRSC always achieve the best performance compared to other competing methods. We employ the fast deep neural network named CNN-F (Chatfield et al., 2014) trained on the ILSVRC 2012 data to extract the 4096-dimensional feature vector for each image in the CIFAR-10 data, and all the clustering methods are performed on the extracted features. In addition to the coordinate descent method employed in Section 2.1.2 and Algorithm 1 for the optimization of the sparse codes in SRSC, we further conduct the empirical study showing that the parallel coordinate descent method, which updates the coordinates in parallel for improved efficiency and fits the needs of large-scale data optimization, leads to almost the same results as the coordinate descent method on the CIFAR-10 data. Instead of optimization with respect to the sparse code of a single data point in the coordinate descent method, the parallel coordinate descent method updates the sparse codes of P data points in parallel using the same rule as that in the coordinate descent method in Section 2.1.2 and Algorithm 1. While the parallel coordinate descent method is originally designed for convex problems (Bradley et al., 2011; Richtárik & Takáč, 2016), it demonstrates almost the same empirical performance as the coordinate descent method for the clustering task on the test set of the CIFAR-10 data, with the accuracy of 0.4622 and NMI of 0.3864. P -parallel coordinate descent leads to P times speedup compared to the coordinate descent method. We choose P = 10 and the codes of the training data of CIFAR-10 are learned by the parallel coordinate descent method, and note that the optimization of the codes of the test data are inherently parallelizable due to the nature of the second test setting studied in this subsection.
Moreover, Table 10 shows the prediction error of Deep-SRSC on the MNIST data and the CIFAR-10 data. It can be observed again that deeper Deep-SRSC network leads to smaller prediction error.
We also show the application to semi-supervised learning via label propagation (Zhu et al., 2003), a widely used semi-supervised learning method. Given the data {x1,x2, . . . ,xl,xl+1, . . . ,xn} ⊂ IRd, the first l points {x1,x2, . . . ,xl} are labeled and named the training data, and the remaining n − l points form the test data for semi-supervised learning. Semi-supervised learning by label propagation aims to predict the labels of the test data by encouraging local smoothness of the labels in accordance with the similarity matrix over the entire data. The performance of label propagation depends on the similarity matrix. For each sparse coding based method, the similarity matrix Y over the entire data is built by the support similarity introduced in Section 5.1: Yij = Aijq
> Ẑi q Ẑj , and Zi is the code of data point xi for different sparse coding methods including the 6-layer Deep-SRSC with the second test setting. Label propagation is performed on the similarity matrix Y to obtain the labels of the test data, and the error rate is reported. Note that in the experiment of semi-supervised learning by label propagation, the codes of the test data of each data set are obtained first (e.g. the 10000 test images in the MNIST data). If xi belongs to the test data of a data set, its code is obtained by performing the the corresponding sparse coding optimization with the dictionary learned on the training data of that data set; for SRSC and Deep-SRSC, such optimization also has the KNN graph over the test point xi and the training data as input. With the codes of all the data, the similarity matrix Y over the entire data is constructed. Then, a randomly sampled subset of each class is labeled as the training data, with the other data serving as the test data for semi-supervised learning.
The semi-supervised learning results of our methods are compared to that of the Gaussian kernel graph (Gaussian), i.e. the KNN graph with the edge weight set by the Gaussian kernel; Sparse Coding (SC) and `2-RSC; and manifold based similarity adaptation (MBS) by Karasuyama & Mamitsuka (2013), one of the state-of-the-art semi-supervised learning methods based on label propagation. MBS learns the manifold aligned edge similarity by local reconstruction for label propagation.
The comparison results of semi-supervised learning by label propagation on the USPS data and the MNIST data are shown in Figure 6 and 7, which illustrate the error rate of label propagation with respect to different number of labeled data points in each class. We can observe from Figure 6 that SRSC and Deep-SRSC with the second test setting lead to superior results on the application to semi-supervised learning, and the performance of SRSC and Deep-SRSC is always the best with respect to different dictionary size. It can also be observed from Figure 6 and 7 that SRSC and Deep-SRSC have very similar performance, revealing the good quality of the fast approximation by Deep-SRSC on the semi-supervised learning task. Furthermore, SRSC and Deep-SRSC significantly outperform other baseline methods with the small number of labeled data points in each class, due to the captured locally linear manifold structure.
Table 9: Prediction error (average squared error between the predicted codes and the ground truth codes) of Deep-SRSC with different depth and different dictionary size on the test set of the USPS data, using the second test setting
Dictionary Size 1-layer 2-layer 6-layer p = 100 0.05 0.03 0.03 p = 300 0.12 0.08 0.06 p = 500 0.27 0.11 0.09
Table 10: Prediction error (average squared error between the predicted codes and the ground truth codes) of Deep-SRSC with different depth on the test set of the MNIST data and CIFAR-10 data, using the second test setting
Data Set 1-layer 2-layer 6-layer MNIST 0.15 0.12 0.10
CIFAR-10 0.16 0.13 0.13

In this paper the authors propose a method to explicitly regularize sparse coding to encode neighbouring datapoints with similar sets of atoms from the dictionary by clustering training examples with KNN in input space. The resulting algorithm is relatively complex and computationally relatively expensive, but the authors provide detailed derivations and use arguments from proximal gradient descent methods to prove convergence (I did not follow all the derivations, only some). In general the paper is well written and the authors explain the motivation behind the algorithms design in detail.
In the abstract the authors mention “extensive experimental results …”, but I find the experiments not very convincing: With experiments on the USPS handwritten digits dataset (why not MNIST?), COIL-20 and COIL-100 and UCI, the datasets are all relatively small and the algorithm is run with dictionary sizes between p=100 to p=500. This seems surprising because the authors state that they implemented SRSC in “CUDA C++ with extreme efficiency” (page 10). But more importantly, I find it hard to interpret and compare the results: The paper reports accuracy and and normalized mutual information for a image retrieval / clustering task where the proposed SRSC is used as a feature extractor. The improvements relative to standard Sparse Coding seem very small  (often < 1% in terms of NMI; it looks more promising in terms of accuracy) and if I understand the description on page 11 correctly, than the test set was used to select some hyperparameters (the best similarity measure for clustering step)? There is no comparisons to other baselines / state of the art image clustering methods.
Besides of providing features for a small scale image clustering system, are there maybe ways to more directly evaluate the properties and qualities of  a sparse coding approach? E.g. reconstruction error / sparsity; maybe even denoising performance?

In summary, I think in it current form the paper lacks the evaluation and experimental results for an ICLR publication. Intuitively, I agree with the authors that the proposed regularization is an interesting direction, but I don’t see experiments that directly show that the regularization has the desired effect; and the improvements in the clustering task where SRSC is used as a feature extractor are very modest.


RECURRENT COEVOLUTIONARY FEATURE EMBEDDING PROCESSES FOR RECOMMENDATION
Recommender systems often use latent features to explain the behaviors of users and capture the properties of items. As users interact with different items over time, user and item features can influence each other, evolve and co-evolve over time. To accurately capture the fine grained nonlinear coevolution of these features, we propose a recurrent coevolutionary feature embedding process model, which combines recurrent neural network (RNN) with a multi-dimensional point process model. The RNN learns a nonlinear representation of user and item embeddings which take into account mutual influence between user and item features, and the feature evolution over time. We also develop an efficient stochastic gradient algorithm for learning parameters. Experiments on diverse real-world datasets demonstrate significant improvements in user behavior prediction compared to state-of-the-arts.
1 INTRODUCTION
E-commerce platforms and social service websites, such as Reddit, Amazon, and Netflix, attracts thousands of users every second. Effectively recommending the appropriate service items to users is a fundamentally important task for these online services. It can significantly boost the user activities on these sites and leads to increased product purchases and advertisement clicks.
The interactions between users and items play a critical role in driving the evolution of user interests and item features. For example, for music streaming services, a long-time fan of Rock music listens to an interesting Blues one day, and starts to listen to more Blues instead of Rock music. Similarly, a single music may also serve different audiences at different times,e.g., a music initially targeted for an older generation may become popular among the young, and the features of this music need to be updated. Furthermore, as users interact with different items, users’ interests and items’ features can also co-evolve over time, i.e., their features are intertwined and can influence each other:
• User → item. In online discussion forums such as Reddit, although a group (item) is initially created for statistics topics, users with very different interest profiles can join this group. Hence, the participants can shape the features of the group through their postings. It is likely that this group can finally become one about deep learning because most users concern about deep learning. • Item→ user. As the group is evolving towards topics on deep learning, some users may become more interested in deep learning topics, and they may participate in other specialized groups on deep learning. On the opposite side, some users may gradually gain interests in pure math groups, lose interests in statistics and become inactive in this group.
Such co-evolutionary nature of user-item interactions raises very important questions on how to learn them from the increasingly available data. However, existing methods either treat the temporal user-item interactions data as a static graph or use epoch based methods such as tensor factorization to learn the latent features (Chi & Kolda, 2012; Koren, 2009; Yang et al., 2011). These methods are not able to capture the fine grained temporal dynamics of user-item interactions. Recent point process based models treat time as a random variable and improves over the traditional methods significantly (Du et al., 2015; Wang et al., 2016b). However, these works make strong assumptions
∗Authors have equal contributions.
about the function form of the generative processes, which may not reflect the reality or accurate enough to capture the complex and nonlinear user-item influence in real world.
In this paper, we propose a recurrent coevolutionary feature embedding process framework. It combines recurrent neural network (RNN) with point process models, and efficiently captures the co-evolution of user-item features. Our model can automatically find an efficient representation of the underlying user and item latent feature without assuming a fixed parametric forms in advance. Figure 1 summarizes our framework. In particular, our work makes the following contributions:
• Novel model. We propose a novel model that captures the nonlinear co-evolution nature of users’ and items’ embeddings. It assigns an evolving feature embedding process for each user and item, and the co-evolution of these latent feature processes is modeled with two parallel components: (i) item→ user component, a user’s latent feature is determined by the nonlinear embedding of latent features of the items he interacted with; and (ii) user→ item component, an item’s latent features are also determined by the latent features of the users who interact with the item. • Technical Challenges. We use RNN to parametrize the interdependent and intertwined user and item embeddings. The increased flexibility and generality further introduces technical challenges on how to train RNN on the co-evolving graphs. The co-evolution nature of the model makes the samples inter-dependent and not identically distributed, which is contrary to the assumptions in the traditional setting and significantly more challenging. We are the first to propose an efficient stochastic training algorithm that makes the BTPP tractable in the co-evolving graph. • Strong performance. We evaluate our method over multiple datasets, verifying that our method can lead to significant improvements in user behavior prediction compared to previous state-of-thearts. Precise time prediction is especially novel and not possible by most prior work.
2 RELATED WORK
Recent work predominantly fix the latent features assigned to each user and item (Salakhutdinov & Mnih, 2008; Chen et al., 2009; Agarwal & Chen, 2009; Ekstrand et al., 2011; Koren & Sill, 2011; Yang et al., 2011; Yi et al., 2014; Wang & Pal, 2015). In more sophisticated methods, the time is divided into epochs, and static latent feature models are applied to each epoch to capture some temporal aspects of the data (Koren, 2009; Karatzoglou et al., 2010; Xiong et al., 2010; Karatzoglou et al., 2010; Xiong et al., 2010; Chi & Kolda, 2012; Gultekin & Paisley, 2014; Charlin et al., 2015; Preeti Bhargava, 2015; Gopalan et al., 2015; Hidasi & Tikk, 2015; Wang et al., 2016a). For such methods, it is not clear how to choose the epoch length parameter. First, different users may have very different timescale when they interact with those service items, making it difficult to choose a unified epoch length. Second, it is not easy for these methods to answer time-sensitive queries such as when a user will return to the service item. The predictions are only in the resolution of the chosen epoch length. Recently, (Du et al., 2015) proposed a low-rank point process based model for time-sensitive recommendations from recurrent user activities. However, it fails to capture the heterogeneous coevolutionary properties of user-item interactions. Wang et al. (2016b) models the co-evolutionary property, but uses a simple linear representation of the users’ and items’ latent features, which might not be expressive enough to capture the real world patterns. As demonstrated in Du et al. (2016),
the nonlinear RNN is quite flexible to approximate many point process models. Also we will show that, our model only has O(#user + #item) regardless of RNN related parameters, and can also be potentially applied to online setting.
In the deep learning community, (Wang et al., 2015a) proposed a hierarchical Bayesian model that jointly performs learning for the content features and collaborative filtering for the ratings matrix. (Hidasi et al., 2016) applied RNN and adopt item-to-item recommendation approach with session based data. (Tan et al., 2016) improved this model with techniques like data augmentation, temporal change adaptation. (Ko et al., 2016) proposed collaborative RNN that extends collaborative filtering method to capture history of user behavior. Specifically, they used static global latent factors for items and assign separate latent factors for users that are dependent on their past history. (Song et al., 2016) extended the deep semantic structured model to capture multi-granularity temporal preference of users. They use separate RNN for each temporal granularity and combine them with feed forward network which models users’ and items’ long term static features. However, none of these works model the coevolution of users’ and items’ latent features and are still extensions of epoch based methods. Our work is unique since we explicitly treat time as a random variable and captures the coevolution of users’ and items’ latent features using temporal point processes. Finally, our work is inspired from the recurrent marked temporal point process model (Du et al., 2016). However, this work only focuses on learning a one-dimension point process. Our work is significantly different since we focus on the recommendation system setting with the novel idea of feature coevolution and we use multi-dimensional point processes to capture user-item interactions.
3 BACKGROUND ON TEMPORAL POINT PROCESSES
A temporal point process (Cox & Isham, 1980; Cox & Lewis, 2006; Aalen et al., 2008) is a random process whose realization consists of a list of discrete events localized in time, {ti} with ti ∈ R+. Equivalently, a given temporal point process can be represented as a counting process, N(t), which records the number of events before time t. An important way to characterize temporal point processes is via the conditional intensity function λ(t), a stochastic model for the time of the next event given all the previous events. Formally, λ(t)dt is the conditional probability of observing an event in a small window [t, t+dt) given the historyH(t) up to t and that the event has not happen before t, i.e.,
λ(t)dt := P {event in [t, t+ dt)|H(t)} = E[dN(t)|H(t)], where one typically assumes that only one event can happen in a small window of size dt, i.e., dN(t) ∈ {0, 1}. Then, given a time t > 0, we can also characterize the conditional probability that no event happens during [0, t) as: S(t) = exp ( − ∫ t 0 λ(τ) dτ ) and the conditional density that an event occurs at time t is defined as f(t) = λ(t)S(t) (1)
The function form of the intensity λ(t) is often designed to capture the phenomena of interests. Some commonly used form includes:
• Hawkes processes (Hawkes, 1971; Wang et al., 2016c), whose intensity models the mutual excitation between events, i.e., λ(t) = µ + α ∑ ti∈H(t) κω(t − ti), where κω(t) := exp(−ωt)
is an exponential triggering kernel, µ > 0 is a baseline intensity. Here, the occurrence of each historical event increases the intensity by a certain amount determined by the kernel κω and the weight α > 0, making the intensity history dependent and a stochastic process by itself. • Rayleigh process, whose intensity function is λ(t) = αt, where α > 0 is the weight parameter.
4 RECURRENT COEVOLUTIONARY FEATURE EMBEDDING PROCESSES
In this section, we present the generative framework for modeling the temporal dynamics of user-item interactions. We first use RNN to explicitly capture the co-evolving nature of users’ and items’ latent feature. Then, based on the compatibility between the users’ and items’ latent feature, we model the user-item interactions by a multi-dimensional temporal point process. We further parametrize the intensity function by the compatibility between users’ and items’ latent features.
4.1 EVENT REPRESENTATION
Given m users and n items, we denote the ordered list of N observed events as O = {ej = (uj , ij , tj , qj)}Nj=1 on time window [0, T ], where uj ∈ {1, . . . ,m}, ij ∈ {1, . . . , n}, tj ∈ R+, 0 6 t1 6 t2 . . . 6 T . This represents the interaction between user uj , item ij at time tj , with the interaction context qj ∈ Rd. Here qj can be a high dimension vector such as the text review, or
simply the embedding of static user/item features such as user’s profile and item’s categorical features. For notation simplicity, we defineOu = {euj = (iuj , tuj , quj )} as the ordered listed of all events related to user u, and Oi = {eij = (uij , tij , qij)} as the ordered list of all events related to item i. We also set ti0 = t u 0 = 0 for all the users and items. tk− denotes the time point just before time tk.
4.2 RECURRENT FEATURE EMBEDDING PROCESSES
We associate feature embeddings uu(t) ∈ Rk with each user u and ii(t) ∈ Rk with each item i. These features represent the subtle properties which cannot be directly observed, such as the interests of a user and the semantic topics of an item. Specifically, we model the drift, evolution, and co-evolution of uu(t) and ii(t) as a piecewise constant function of time that has jumps only at event times. Specifically, we define:
User latent feature embedding process. For each user u, the corresponding embedding after user u’s k-th event euk = (i u k , t u k , q u k ) can be formulated as:
uu(t u k) = σ ( W1(t
u k − tuk−1)︸ ︷︷ ︸
temporal drift
+W2uu(t u k−1)︸ ︷︷ ︸
self evolution
+ W3iik(t u k−)︸ ︷︷ ︸
co-evolution: item feature
+ W4q u,ik k︸ ︷︷ ︸
interaction feature
) (2)
Item latent feature embedding process. For each item i, we specify ii(t) at time tik as:
ii(t i k) = σ ( V1(t
i k − tik−1)︸ ︷︷ ︸
temporal drift
+V2ii(t i k−1)︸ ︷︷ ︸
self evolution
+ V3uuk(t i k−)︸ ︷︷ ︸
co-evolution: item feature
+ V4q i,uk k︸ ︷︷ ︸
interaction feature
) (3)
where t− means the time point just before time t, W4,V4 ∈ Rk×d are the embedding matrices mapping from the explicit high-dimensional feature space into the low-rank latent feature space and W1,V1 ∈ Rk, W2,V2,W3,V3 ∈ Rk×k are weights parameters. σ(·) is the nonlinear activation function, such as commonly used Tanh or Sigmoid for RNN. For simplicity, we use basic recurrent neural network to formulate the recurrence, but it is also straightforward to extend it using GRU or LSTM to gain more expressive power. Figure 1 summarizes the basic setting of our model.
Here both the user and item’s feature embedding processes are piecewise constant functions of time and only updated if an interaction event happens. A user’s attribute changes only when he has a new interaction with some item. For example, a user’s taste for music changes only when he listens to some new or old musics. Also, an item’s attribute changes only when some user interacts with it. Different from Chen et al. (2013) who also models the time change with piecewise constant function, but their work has no coevolve modeling, and is not capable of predicting the future time point.
Next we discuss the rationale of each term in detail:
• Temporal drift. The first term is defined based on the time difference between consecutive events of specific user or item. It allows the basic features of users (e.g., a user’s self-crafted interests) and items (e.g., textual categories and descriptions) to smoothly drift through time. Such changes of basic features normally are caused by external influences. • Self evolution. The current user feature should also be influenced by its feature at the earlier time. This captures the intrinsic evolution of user/item features. For example, a user’s current taste should be more or less similar to his/her tastes two days ago. • User-item coevolution. Users’ and items’ latent features can mutually influence each other. This term captures the two parallel processes. First, a user’s embedding is determined by the latent features of the items he interacted with. At each time tk, the latent item feature is iik(t u k−).
We capture both the temporal influence and feature of each history item as a latent embedding. Conversely, an item’s embedding is determined by the feature embedding of the user who just interacts with the item. • Evolution with interaction features. Users’ and items’ features can evolve and be influenced by the characteristics of their interactions. For instance, the genre changes of movies indicate the changing tastes of users. The theme of a chatting-group can be easily shifted to certain topics of the involved discussions. In consequence, this term captures the influence of the current interaction features to the changes of the latent user (item) features. • Interaction feature. This is the additional information happened in the user-item interactions. For example, in online discussion forums such as Reddit, the interaction features are the posts and comments. In online review sites such as Yelp, it is the reviews of the businesses.
To summarize, each feature embedding process evolves according to the respective base temporal user (item) features and also are mutually dependent on each other due to the endogenous influences from the interaction features and the entangled latent features.
4.3 USER-ITEM INTERACTIONS AS TEMPORAL POINT PROCESSES
For each user, we model the recurrent occurrences of all users interaction with all items as a multidimensional temporal point process, with each user-item pair as one dimension. In particular, the intensity function in the (u, i)-th dimension (user u and item i) is modeled as a Rayleigh process:
λu,i(t|t′) = exp ( uu(t ′)>ii(t ′) )︸ ︷︷ ︸
user-item compatibility
· (t− t′)︸ ︷︷ ︸ time lapse
(4)
where t > t′, and t′ is the last time point where either user u’s embedding or item i’s embedding changes before time t. The rationale behind this formulation is three-fold:
• Time as a random variable. Instead of discretizing the time into epochs as traditional methods (Charlin et al., 2015; Preeti Bhargava, 2015; Gopalan et al., 2015; Hidasi & Tikk, 2015; Wang et al., 2016a), we explicitly model the timing of each interaction event as a random variable, which naturally captures the heterogeneity of the temporal interactions between users and items. • Short term preference. The probability for user u to interact with item i depends on the compatibility of their instantaneous embeddings, which is evaluated through the inner product at the last event time t′. Because uu(t) and ii(t) co-evolve through time, their inner-product measures a general representation of the cumulative influence from the past interactions to the occurrence of the current event. The exp(·) function ensures the intensity is positive and well defined. • Rayleigh time distribution. The user and item embeddings are piecewise constant, and we use the time lapse term to make the intensity piecewise linear. This form leads to a Rayleigh distribution for the time intervals between consecutive events in each dimension. It is well-adapted to modeling fads, where the event-happening likelihood f(·) in (1) rises to a peak and then drops extremely rapidly. Furthermore, it is computationally easy to obtain an analytic form of f(·). One can then use f(·) to make item recommendation by finding the dimension that f(·) reaches the peak.
With the parameterized intensity function, we can further estimate the parameters using maximum likelihood estimation of all events. The joint negative log-likelihood is (Daley & Vere-Jones, 2007):
` = − N∑ j=1 log ( λuj ,ij (tj |t′j) ) ︸ ︷︷ ︸
intensity of interaction event
+ m∑ u=1 n∑ i=1 ∫ T 0
λu,i(τ |τ ′) dτ︸ ︷︷ ︸ survival probability of event not happened
(5)
The rationale of the objective two-fold: (i) the negative intensity summation term ensures the probability of all interaction events is maximized; (ii) the second survival probability term penalizes the non-presence of an interaction between all possible user-item pairs on the observation window. Hence, our framework not only explains why an event happens, but also why an event did not happen.
5 PARAMETER LEARNING
In this section, we propose an efficient algorithm to learn the parameters {Vi}4i=1 and {Wi} 4 i=1. The batch objective function is presented in (5). The Back Propagation Through Time (BPTT) is the standard way to train a RNN. To make the back propagation tractable, one typically needs to do truncation during training. However, due to the novel co-evolutionary nature of our model, all the events are related to each other by the user-item bipartite graph (Figure 2), which makes it hard to decompose.
Hence, in sharp contrast to works (Hidasi et al., 2016; Du et al., 2016) in sequential data where one can easily break the sequences into multiple segments to make the BPTT trackable, it is a challenging task to design BPTT in our case. To efficiently solve this problem, we first order all the events globally and then do mini-batch training in a sliding window fashion. Each time when conducting feed forward and back propagation, we take the consecutive events within current sliding window to build the computational graph. Thus in our case the truncation is on the global timeline, instead over individual independent sequences as in prior works.
Next, we explain our procedure in detail. Given a mini-batch of M ordered events Õ = {ej}Mj=1, we set the time span to be [T0 = t1, T = tM ]. Below we show how to compute the intensity and survival probability term in the objective function (5) respectively.
embedding is shown. (b) Survival probability for a user-item pair (u, i). The integral
∫ T
0 λu,i(τ |τ ′)dτ is decomposed into 4 inter-event intervals separated by {t0, · · · , t3}, with close form on each interval.
Computing the intensity function. Each time when a new event ej happens between uj and ij , their corresponding feature embeddings will evolve according to a computational graph, as illustrated in Figure 2a. Due to the change of feature embedding, all the dimensions related to uj or ij will be influenced and the intensity function for that dimension will change consequently. Such crossdimension influence dependency is shown in Figure 2b. In our implementation, we first compute the corresponding intensity λuj ,ij (tj |t′j) according to (4), and then update the embedding of uj and ij . This operation takes O(M) complexity, and is independent to the number of users or items.
Computing the survival function. To compute the survival probability − ∫ T T0 λu,i(τ |τ ′)dτ for each pair (u, i), we first collect all the time stamps {tk} that have events related to either u or i. For notation simplicity, let |{tk}| = nu,i and t1 = T0, tnu,i = T . Since the embeddings are piecewise constant, the corresponding intensity function is piecewise linear, according to (4). Thus, the integration is decomposed into each time interval where the intensity is constant, i.e.,∫ T
T0 λu,i(τ |τ ′)dτ = nu,i−1∑ k=1 ∫ tk+1 tk λu,i(τ |τ ′)dτ = nu,i−1∑ k=1 (t2k+1 − t2k) exp ( uu(tk) >ii(tk) ) (6)
Figure 3 visualizes the computation. Although the survival probability term exists in close form, we still need to solve two challenges. First, it is still expensive to compute it for each user item pair. Moreover, since the user-item interaction bipartite graph is very sparse, it is not necessary to monitor each dimension in the stochastic training setting. To speed up the computation, we propose a novel random-sampling scheme as follows.
Note that the intensity term in the objective function (5) tries to maximize the inner product between user and item that has interaction event, while the survival term penalize over all other pairs of inner
products. We observe that this is similar to Softmax computing for classification problem. Hence, inspired by the noise-contrastive estimation method (Gutmann & Hyvärinen, 2012) that is widely used in language models (Mnih & Kavukcuoglu, 2013), we keep the dimensions that have events on them, while randomly sample dimensions without events in current mini-batch.
The second challenge lies in the fact that the user-item interactions vary a lot across mini-batches, hence the corresponding computational graph also changes greatly. To make the learning efficient, we use the graph embedding framework (Dai et al., 2016) which allows training deep learning models where each term in the objective has a different computational graphs but with shared parameters. The Adam Optimizer (Kingma & Ba, 2014) together with gradient clip is used in our experiment.
6 EXPERIMENTS
We evaluate our model on real-world datasets. For each sequence of user activities, we use all the events up to time T · p as the training data, and the rest events as the testing data, where T is the observation window. We tune the latent rank of other baselines using 5-fold cross validation with grid search. We vary the proportion p ∈ {0.7, 0.72, 0.74, 0.76, 0.78} and report the averaged results over five runs on two tasks (we will release code and data once published):
• Item prediction. At each test time t, we predict the item that the user u will interact with. We rank all the items in the descending order of the conditional density fu,i(t) = λu,i(t)Su,i(t). We report the Mean Average Rank (MAR) of each test item at the test time. Ideally, the item associated with the test time t should rank one, hence smaller value indicates better predictive performance. • Time prediction. We predict the expected time when a testing event will occur between a given user-item pair. Using Rayleigh distribution, it is given by Et∼fu,i(t)(t) = √ π
2 exp(uu(t−)>ii(t−)) .
We report the Mean Absolute Error (MAE) between the predicted and true time.
6.1 COMPETITORS
We compared our DEEPCOEVOLVE with the following methods. Table 1 summarizes the differences.
• LowRankHawkes (Du et al., 2015): This is a low rank Hawkes process model which assumes user-item interactions to be independent of each other and does not capture the co-evolution of user and item features. • Coevolving (Wang et al., 2016b): This is a multi-dimensional point process model which uses a simple linear embedding to model the co-evolution of user and item features. • PoissonTensor (Chi & Kolda, 2012): Poisson Tensor Factorization has been shown to perform better than factorization methods based on squared loss (Karatzoglou et al., 2010; Xiong et al., 2010; Wang et al., 2015b) on recommendation tasks. The performance for this baseline is reported using the average of the parameters fitted over all time intervals. • TimeSVD++ (Koren, 2009) and FIP (Yang et al., 2011): These two methods are only designed for explicit ratings, the implicit user feedbacks (in the form of a series of interaction events) are converted into the explicit ratings by the respective frequency of interactions with users. • STIC (Kapoor et al., 2015): it fits a semi-hidden markov model (HMM) to each observed user-item pair and is only designed for time prediction.
6.2 DATASETS
We use three real world datasets as follows.
• IPTV. It contains 7,100 users’ watching history of 385 TV programs in 11 months (Jan 1 - Nov 30 2012), with around 2M events, and 1,420 movie features (including 1,073 actors, 312 directors, 22 genres, 8 countries and 5 years). • Yelp. This data was available in Yelp Dataset challenge Round 7. It contains reviews for various businesses from October, 2004 to December, 2015. The dataset we used here contains 1,005 users and 47,924 businesses, with totally 291,716 reviews.
• Reddit. We collected discussion related data on different subreddits (groups) for the month of January 2014. We filtered all bot users’ and their posts from this dataset. Furthermore, we randomly selected 1,000 users, 1,403 groups, and 10,000 discussion events.
6.3 PREDICTION RESULTS
Figure 4 shows that DEEPCOEVOLVE significantly outperforms both epoch-based baselines and state-of-arts point process based methods. LOWRANKHAWKES has good performance on item prediction but not on time prediction, while COEVOLVING has good performance on time prediction but not on item prediction. We discuss the performance regarding the two metrics below.
Item prediction. Note that the best possible MAR one can achieve is 1, and our method gets quite accurate results: with the value of 1.7 on IPTV and 1.9 on Reddit. Note LOWRANKHAWKES achieves comparable item prediction performance, but not as good on the time prediction task. We think the reason is as follows. Since one only need the rank of conditional density f(·) in (1) to conduct item prediction, LOWRANKHAWKES may still be good at differentiating the conditional density function, but could not learn its actual value accurately, as shown in the time prediction task where the value of the conditional density function is needed for precise prediction.
Time prediction. The second row of Figure 4 shows that DEEPCOEVOLVE outperforms other methods. Compared with LOWRANKHAWKES that achieves comparable time predication performance, 6× improvement on Reddit, it has 10× improvement on Yelp, and 30× improvement on IPTV. The time unit is hour. Hence it has 2 weeks accuracy improvement on IPTV and 2 days on Reddit. This is important for online merchants to make time sensitive recommendations. An intuitive explanation is that our method accurately captures the nonlinear pattern between user and item interactions. The competitor LOWRANKHAWKES assumes specific parametric forms of the user-item interaction process, hence may not be accurate or expressive enough to capture real world temporal patterns. Furthermore, it models each user-item interaction dimension independently, which may lose the important affection from user’s interaction with other items while predicting the current item’s reoccurrence time. Our work also outperforms COEVOLVING, e.g., with around 3×MAE improve on IPTV. Moreover, the item prediction performance is also much better than COEVOLVING. It shows the importance of using RNN to capture the nonlinear embedding of user and item latent features, instead of the simple parametrized linear embedding in COEVOLVING.
6.4 INSIGHT OF RESULTS
We will look deeper and provide rationale behind the prediction results in the following two subsections. First, to understand the difficulty of conducting prediction tasks in each dataset, we study their different sparsity properties. For the multidimensional point process models, the fewer events we observe in each dimension, the more sparse the dataset is. Our approach alleviates the sparsity problem via the modeling of dependencies among dimensions, thus is consistently doing better than other baseline algorithms.
Next, we fix one dataset and evaluate how different levels of sparsity in training data influences each algorithm’s performance.
6.4.1 UNDERSTANDING THE DATASETS
We visualize the three datasets in Figure 5 according to (i) the number of events per user, and (ii) the user-item interaction graph.
Sparsity in terms of the number of events per user. Typically, the more user history data we have, the better results we will obtain in the prediction tasks. We can see in IPTV dataset, users typically have longer length of history than the users in Reddit and Yelp datasets. Thus our algorithm and all other baseline methods have their best performance on this dataset. However, the Reddit dataset and Yelp dataset are hard to tell the performance based only on the distribution of history length, thus we do a more detailed visualization.
Sparsity in terms of diversity of items to recommend. From the bipartite graph, it is easy to see that Yelp dataset has higher density than the other two datasets. The density of the interaction graph reflects the variety of history per each user. For example, the users in IPTV only has 385 programs to watch, but they can have 47,924 businesses to choose in Yelp dataset. Also, the Yelp dataset has 9 times more items than IPTV and Reddit dataset in the bipartite graph. This means the users in Yelp dataset has more diverse tastes than users in other two datasets. This is because if users has similar tastes, the distinct number of items in the union of their history should be small.
Based on the above two facts, we can see Yelp dataset is the most sparse, since it has shorter length of history per user, and much more diversity of the items, it is not surprising that this dataset is much harder than the other IPTV and Reddit dataset.
6.4.2 ROBUSTNESS OF THE ALGORITHM
With the case study on the most challenging Yelp dataset, we further evaluate how each algorithm performs with lower level of sparsity as compared to the one used in Figure 4 (c).We use this to demonstrate that our work is most robust and performs well across different levels of sparsity.
We first create Yelp100, a more dense dataset, by filtering the original Yelp dataset to keep the top 100 users. Each user would have at least 200 events. Figure 6 (a) shows the statistics of this dataset. On average the users have more history events than the original Yelp dataset in Figure 5(c).
On this dense dataset, Figure 6 (b) and (c) show that all the algorithms’ performances improve with more history events, comparing to the performance in original Yelp dataset. For example, LOWRANKHAWKES has similar rank prediction results as our DEEPCOEVOLVE on this dense dataset. However, as the dataset becomes sparse, the performance of LOWRANKHAWKES drops significantly, as shown in Figure 4(c). For example, the rank prediction error goes from 90 to 2128, and the
time error goes from 724 to 11043.5. We think it is because this model relies more on the history information per each user-item pair.
On the contrary, our DEEPCOEVOLVE still has superior performance with such high level of sparsity. The rank error only changes from 87 to 107, and the time error changes from 72 to 884 as the data becomes sparse. It shows that our work is the most robust to the sparsity in the data. We think it is because our work accurately captures the nonlinear multidimensional dependencies between users and items latent features.
7 CONCLUSION
We have proposed an efficient framework to model the nonlinear co-evolution nature of users’ and items’ latent features. Moreover, the user and item’s evolving and co-evolving processes are captured by the RNN. It is based on temporal point processes and models time as a random variable. Hence it is in sharp contrast to prior epoch based works. We demonstrate the superior performance of our method on both the time and item prediction task, which is not possible by most prior work. Future work includes extending to other social applications, such as group dynamics in message services.
A DETAILS ON GRADIENT COMPUTATION
Computing gradient. For illustration purpose, we here use Sigmoid as the nonlinear activation function σ. In order to get gradient with respect to parameter W ’s, we first compute gradients with respect to each varying points of embeddings. For user u’s embedding after his k-th event, the corresponding partial derivatives are computed by:
∂`
∂uu(tuk) = −iiuk︸︷︷︸
from intensity
+ n∑ i=1
∂ ∫ tuk+1 tuk
λu,i(τ |τ ′)dτ ∂uu(tuk)︸ ︷︷ ︸
from survival
+ ∂`
∂uu(tuk+1) (1− uu(tuk+1)) uu(tuk+1)W2︸ ︷︷ ︸ from user u’s next embedding
+ ∂`
∂iiuk+1(t u k+1)
(1− iiuk+1(t u k+1)) iiuk+1(t u k+1)︸ ︷︷ ︸
from user u’s next item embedding
where denotes element-wise multiplication. The gradient coming from the second term (i.e., the survival term) is also easy to compute, since the Rayleigh distribution has closed form of survival function. For a certain item i, if its feature doesn’t changed between time interval [tuk , t u k+1], then we have
∂ ∫ tuk+1 tuk
λu,i(τ |τ ′)dτ ∂uu(tuk) = (tuk+1 − tuk)2 2 exp
( uu(t u k) >ii(t u k)ii(t u k) )
(7)
On the other hand, if the embedding of item i changes during this time interval, then we should break this interval into segments and compute the summation of gradients in each segment in a way similar to (7). Thus, we are able to compute the gradients with respect to Wi, i ∈ {1, 2, 3, 4} as follows.
∂`
∂W1 = m∑ u=1 ∑ k
∂`
∂uu(tuk) (i− uu(tuk)) uu(tuk)(tuk − tuk−1)
∂`
∂W2 = m∑ u=1 ∑ k ( ∂` ∂uu(tuk) (i− uu(tuk)) uu(tuk) ) uu(t u k−1) >
∂`
∂W3 = m∑ u=1 ∑ k ( ∂` ∂uu(tuk) (i− uu(tuk)) uu(tuk) ) iik(t u k−)>
∂`
∂W4 = m∑ u=1 ∑ k ( ∂` ∂uu(tuk) (i− uu(tuk)) uu(tuk) ) qu,ikk
Since the items are treated symmetrically as users, the corresponding derivatives can be obtained in a similar way.

The paper introduces a time dependent recommender system based on point processes parametrized by time dependent user and item latent representations. The later are modeled as coupled – autoregressive processes – i.e. the representation of a user/item changes when he interacts with an item/user, and is a function of both the user and the item representations before time t. This is called coevolution here and the autoregressive process is called recurrent NN. The model may also incorporate heterogeneous inputs. Experiments are performed on several datasets, and the model is compared with different baselines.
There are several contributions in the paper: 1) modeling recommendation via parametrized point processes where the parameter dynamics are modeled by latent user/item representations, 2) an optimization algorithm for maximizing the likelihood of this process, with different technical tricks that seem to break its intrinsic complexity, 3) evaluation experiments for time dependent recommendation.
The paper by the same authors (NIPS 2016) describes a similar model of continuous time coevolution, and a similar evaluation. The difference lies in the details of the model: the point process model is not the same and of the latent factor dynamic model is slightly different, but the modeling approach and the arguments are exactly the same. By the end, one does not know what makes this model perform better than the one proposed in NIPS, is it the choice for the process, the new parametrization? Both are quite similar. There is no justification on the choice of the specific form of the point process in the two papers. Did the authors tried other forms as well? The same remark applies for the form of the dynamical process: the non-linearity used for the modeling of the latent user/item vectors here is limited to a sigmoid function, which probably does not change much w.r.t. a linear model, but there is no evidence of the role of this non linearity in the paper. Note that there are some inconsistencies between the results in the two papers.
Concerning the evaluation, the authors introduce two criteria. I did not get exactly how they evaluate the item recommendation: it is mentioned that at each time t, the model predicts the item the user will interact with. Do you mean, the next item the user will interact with after time t? For the time prediction, why is it a relevant metric for recommendation?
A comparison of the complexity, or execution time of the different methods would be helpful. The complexity of your method is apparently proportional to #items*#users, what are the complexity limits of your methods.
Overall, the paper is quite nice and looks technically sound, albeit many details are missing. On the other hand, I have a mixed feeling because of the similarity with NIPS paper. The authors should have make a better work at convincing us that this is not a marginal extension of previous work by the authors.  I was not convinced either by the evaluation criteria and there is no evidence that the model can be used for large datasets.