Collaborative filtering (CF) is a core technique for recommender systems. Traditional CF approaches exploit user-item relations (e.g., clicks, likes, and views) only and hence they suffer from the data sparsity issue. Items are usually associated with unstructured text such as article abstracts and product reviews. We develop a Personalized Neural Embedding (PNE) framework to exploit both interactions and words seamlessly. We learn such embeddings of users, items, and words jointly, and predict user preferences on items based on these learned representations. PNE estimates the probability that a user will like an item by two termsbehavior factors and semantic factors. On two real-world datasets, PNE shows better performance than four state-of-the-art baselines in terms of three metrics. We also show that PNE learns meaningful word embeddings by visualization. Recommender systems are widely used in e-commerce platforms, such as to help consumers buy products at Amazon, watch videos on Youtube, and read articles on Google News. They are useful to alleviate the information overload and improve user satisfaction. Given history records of consumers such as the product transactions and movie watching, collaborative filtering (CF) is among the most effective approaches based on the simple intuition that if users rated items similarly in the past then they are likely to rate items similarly in the future (Sarwar et al., 2001). History records include both implicit (e.g., purchase and clicks) and explicit (e.g., likes/dislikes and ratings) feedback which can be represented as a user-item interaction matrix. Typically, observed user-item interactions are incomplete with a large portion remaining not recorded. The goal of recommendation is to predict user preferences on these missing interactions. This setting requires to complete the partial observed rating matrix. Matrix Factorization (MF) techniques which can learn latent factors for users and items are the main cornerstone for CF (Mnih and Salakhutdinov, 2008; Koren, 2008; Koren et al., 2009). It is effective and flexible to integrate with additional data sources (Hu et al., 2015). Recently, neural networks like Multilayer Perceptron (MLP) are used to learn an interaction function from data with the power of learning highly nonlinear relationships between users and items (Dziugaite and Roy, 2015; Cheng et al., 2016; He et al., 2017; Hu et al., 2018b). MF and neural CF exploit user-item behavior interactions only and hence they both suffer from the data sparsity and cold-start issues. Items are usually associated with unstructured text, like news articles and product reviews. These additional sources are essential for recommendation beyond user-item interactions since they contain independent and diverse information. Hence, they provide an opportunity to alleviate the data sparsity issue (Ganu et al., 2009; Hu et al., 2018a). For application domains like recommending research papers and news articles, the unstructured text associated with the item is its text content (Wang and Blei, 2011; Wang et al., 2015; Bansal et al., 2016). For some domains like recommending products, the unstructured text associated with the item is its user reviews which justify the rating behavior (McAuley and Leskovec, 2013; He and McAuley, 2016). These methods adopt topic modelling techniques and neural networks to exploit the item content leading to performance improvement. A typical way of exploiting text content is to firstly extract a feature vector for each document by averaging word embeddings in the document, and then to learn a text factor corresponding to this feature vector (Hu and Dai, 2017). These embeddings are pre-trained from a large corpus such as Wikipedia. This approach separates the extraction of text feature from the learning of user-item interaction. These two processes cannot benefit from each other and errors in the previous step maybe propagate to the successive steps. Another way is to learn a topic vector using topic modelling (Wang and Blei, 2011; McAuley and Leskovec, 2013; Bao et al., 2014) by aligning behavior factors and topic factors with a link function such as softmax and offset. Recently, neural networks are used to learn a representation from the text using autoencoder-s (Wang et al., 2015; Zhang et al., 2016), recurrent networks (Bansal et al., 2016), and convolutional networks (Zheng et al., 2017; Catherine and Cohen, 2017). These methods treat different words in the document as equal importance and do not match word semantics with the specific user. Instead, we achieve to learn a personalized word embedding with the guidance of user-item interactions. That is, the importance of words is learned to match user preferences. The attention mechanism can be used to learn these importance weights. Memory Networks (MemNet) have been used in recommendation to model item content (Hu et al., 2018c; Huang et al., 2017), capture user neighborhood (Ebesu et al., 2018), and learn latent relationships (Tay et al., 2018). We follow this thread to adapt a MemNet to match word semantics with user preferences. In this paper, we propose a novel neural framework to exploit relational interactions and text content seamlessly. The proposed Personalized Neural Embedding (PNE) model fuses semantic representations learnt from unstructured text with behavior representations learnt from user-item interactions jointly for effective estimation of user preferences on items. PNE estimates the preference probability by two kinds of factors. The behavior factor is to capture the personalized preference of a user to an item learned from behavior interactions. The semantic factor is to capture the high-level representation attentively extracted from the unstructured text by matching word semantics with user preferences. To model the behavior factor, we adopt a neural CF approach, which learns the user-item nonlinear interaction relationships using a neural network (CFNet). To model the semantic factor, we adopt a memory network to match word semantics with the specific user via the attention mechanism inherent in the memory module (MemNet), determining which words are highly relevant to the user preferences. PNE integrates relational interactions with unstructured text by bridging neural CF and memory networks. PNE can also learn meaningful word embeddings. We present PNE to jointly learn representations of users, items, and words. PNE seamlessly captures nonlinear user-item interaction relationships and matches word semantics with user preferences. Denote the set of users by U and items by I . We use a rating matrix Y  R |U||I| to describe user-item interactions where each entry y ui  { 0 , 1 } is 1 (observed entries) if user u has an interaction with item i and 0 (unobserved entries) otherwise. Usually the interaction matrix is very sparse since a user u  U only consumed a very small subset of all items. For the task of item recommendation, each user is only interested in identifying topK items (typically K is small e.g. tens or hundreds). Items are ranked by their predicted scores:  y ui = f ( u, i | ) , (1) where f is an interaction function and  denotes model parameters. PNE consists of a CF network (CFNet) to learn a nonlinear interaction function and of a memory network (MemNet) to match word semantics with user preferences. The information flow in PNE goes from the input ( u, i ) to the output  y ui through the following five modules. 1. Input: ( u, i )  ( (cid:126)e u ,(cid:126)e i ) This module encodes user-item interaction indices. We adopt the one-hot encoding. It takes user u and item i , and maps them into one-hot encodings (cid:126)e u  { 0 , 1 } |U| and (cid:126)e i  { 0 , 1 } |I| where only the element corresponding to that user/item index is 1 and all others are 0. 2. Embedding: ( (cid:126)e u ,(cid:126)e i )  x ui This module firstly embeds one-hot encodings into continuous representations x u = PT (cid:126)e u and x i = QT (cid:126)e i by embedding matrices P  R |U| d and Q  R |I| d respectively, where d is the latent dimension. It then concatenates them as x ui = [ x u , x i ] to be the input of following CFNet and MemNet modules. 3. CFNet: x ui  z behavior ui This module is a CF approach to exploit user-item interactions. It takes continuous representations from the embedding Dataset #user #item #rating #word #density avg. module and then transforms to a final behavior factor representation: z behavior ui = ReLU ( W x ui + b ) , (2) where ReLU ( x ) = max(0 , x ) is an activation function, and W and b are connection weights and biases. 4. MemNet: x ui  z semantic ui This module is to model the item content with the guidance of user-item interaction. The item content is modelled by memories. It takes representations from both the embedding module and the review text d ui associated with the corresponding user-item ( u, i ) into a final semantic factor representation: z semantic ui = (cid:88) j : w j  d ui Softmax ( a u,ij ) c j , (3) where the external memory slot c j is an embedding vector for word w j by mapping it with an external memory matrix C . The attentive weight a u,ij encodes the relevance of user u to word w j by content-based addressing: a u,ij = x Tui m u,ij , (4) where memory m u,ij is concatenated from internal memory slots { m uj , m ij } which are mapped from word w j by internal memory matrices A u for user attention and A i for item attention. 5. Output: z ui   y ui This module predicts the recommendation score  y ui for a given user-item pair based on the representation of both behavior factor and semantic factor from CFNet and MemNet respectively: z ui = [ z behavior ui , z semantic ui ] . The output is the probability that the input pair is a positive interaction. This is achieved by a logistic layer:  y ui = 1 1 + exp(  h T z ui ) , (5) where h is model parameter. We adopt the binary cross-entropy loss: L () =  (cid:88) ( u,i ) S y ui log  y ui +(1  y ui ) log(1   y ui ) where S = Y +  Y  is the union of observed interactions and randomly sampled negative examples. Model parameters  = { P , Q , W , b , A , h } where we use a single word embedding matrix A by sharing all memory matrices A u , A i , and C in order to reduce model complexity. The objective function can be optimized by stochastic gradient descent. In this section, we evaluate PNE on two datasets with five baselines in terms of three metrics. We evaluate on two real-world datasets. The public Amazon products (McAuley and Leskovec, 2013) and a company Cheetah Mobile news (Hu et al., 2018c; Liu et al., 2018) (see Table 1). We preprocess the data following the strategy in (Wang and Blei, 2011). The size of word vocabulary is 8,000. We adopt leave-one-out evaluation (Hu et al., 2018b) and use three ranking metrics: hit ratio (HR), normalized discounted cumulative gain (ND-CG), and mean reciprocal rank (MRR). We compare with five baselines (see Table 2). BPR (Rendle et al., 2009) is a latent factor model based on matrix factorization. HFT (McAuley and Leskovec, 2013) adopts topic distributions to learn latent factors from text reviews. TBPR (Hu and Dai, 2017) extends BPR by integrating text content via word embedding features. Word embeddings used in TBPR are pre-trained by GloVe (Pennington et al., 2014). MLP (He et al., 2017) is a neural CF approach. Note that, CFNet of PNE is an MLP with only one hidden layer. LCMR (Hu et al., 2018c) is a deep model for CF with unstructured text. Note that, MemNet of PNE is the same with the local MemNet of LCMR with only one-hop hidden layer. Our method is implemented by TensorFlow (Abadi et al., 2016). Parameters are randomly initialized from Gaussian with optimizer Adam (K-ingma and Ba, 2015). Learning rate is 0.001, batch size is 128, the ratio of negative sampling is 1. 3.3 Results Results on two datasets are shown in Table 3 and Table 4, respectively. We have some observations. First, PNE outperforms the neural CF method MLP on two datasets in terms of three ranking metrics. On Amazon dataset, PNE obtains a large improvement in performance gain with relative 12.3% HR@10, 7.7% NDCG@10, and 6.2% MRR@5. On Cheetah Mobile dataset, PNE obtains a large improvement in performance gain with relative 5.0% HR@5, and 4.2% NDCG@5, and 3.9% MRR@5. Since the CFNet component of PNE is a neural CF method (with only one hidden layer), results show Figure 1: Dimension of embedding. the benefit of exploiting unstructured text to alleviate the data sparsity issue faced by CF methods (BPR and MLP). Second, PNE outperforms the traditional hybrid methods HFT and TBPR on two datasets in terms of three ranking metrics. On Amazon dataset, PNE obtains a significantly large improvement in performance gain with relative 55.0% HR@5, 28.9% NDCG@5, and 20.4% MRR@5. On Cheetah Mobile dataset, PNE still obtains reasonably large improvements with relative 17.5% HR@10, 1.8% NDCG@10, and 1.9% MRR@10. Compared with traditional hybrid methods which integrate the text content using topic modelling or word embeddings, results show the benefit of integrating text information through memory networks (and exploiting the interaction data through neural CF). Last, PNE outperforms neural hybrid method LCMR by a large margin on Amazon dataset with relative improvements of 16.2% HR@5, 9.6% NDCG, and 7.4% MRR@5. PNE obtains reasonable improvements on Cheetah Mobile dataset with relative improvements of 3.1% HR@5, 2.8% NDCG, and 2.7% MRR. The design of CFNet of PNE is Figure 3: Visualization of word embeddings. more reasonable than that of centralized memory module of LCMR which is equivalent to use a softmax activation between two hidden layers. The results show the effectiveness of fusing strategy in PNE to exploit unstructured text via MemNet and the interaction data via CFNet. We first evaluate the effects of the dimensionality of the embedding space. The x -axis in Figure 1 is the dimension of user/item and hence the dimensionality of input to CFNet is double since we adopt concatenation. It clearly indicates that the embedding should not be too small due to the possibility of information loss and the limits of expressiveness. The dimension 75 (and hence d = 150 ) is a good tradeoff between recommendation performance and computation burden. We next show optimization curves of performance and loss (averaged over all examples) against iterations on Cheetah Mobile dataset in Figure 2. The model learns quickly in the first 20 iterations and improves slowly until 50, while training losses continue to go down and valid losses stabilize. The average time per epoch of PNE takes 68.1s and as a reference it is 34.5s for MLP using one NVIDIA TITAN Xp GPU. We visualize the learned word embeddings A . We show that we can learn meaningful semantics for word embeddings such that words are to cluster when they have relevant semantics. We give an example to show the neighbors of the word drug in the 3D space by projecting the high-dimensional word vectors using TensorFlow 1 as shown in Figure 3. The top nearest neighbors of drug are: shot, shoots, gang, murder, killing, rape, stabbed, truck, school, police, teenage . We can see they are highly semantic relevant. We may also infer that school teenagers have close relationships to the drug issue from the Cheetah News corpus. This should raise a concern for society and it shows the society impact of natural language processing (Hovy and Spruit, 2016). Try our trained word embeddings 2 . We showed that relational interactions can be effectively integrated with unstructured text under a neural embedding model. Our method attentively focuses relevant words to match user preferences with user and item attentions (semantic factor) and captures nonlinear relationships between users and items (behavior factor). Experiments show better performance than five baselines on two real-world datasets in terms of three ranking metrics. We learn meaningful word embeddings and rethink the society impact of language processing technology. The work is supported by HK CERG projects 16211214/16209715/16244616, NSFC 61673202, and HKPFS PF15-16701.
abstain abstain abstain objective method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method objective abstain abstain abstain abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain result abstain abstain abstain result method abstain objective other