Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks
Hybrid methods that utilize both content and rating information are commonly used in many recommender systems. However, most of them use either handcrafted features or the bag-of-words representation as a surrogate for the content information but they are neither effective nor natural enough. To address this problem, we develop a collaborative recurrent autoencoder (CRAE) which is a denoising recurrent autoencoder (DRAE) that models the generation of content sequences in the collaborative filtering (CF) setting. The model generalizes recent advances in recurrent deep learning from i.i.d. input to non-i.i.d. (CF-based) input and provides a new denoising scheme along with a novel learnable pooling scheme for the recurrent autoencoder. To do this, we first develop a hierarchical Bayesian model for the DRAE and then generalize it to the CF setting. The synergy between denoising and CF enables CRAE to make accurate recommendations while learning to fill in the blanks in sequences. Experiments on real-world datasets from different domains (CiteULike and Netflix) show that, by jointly modeling the order-aware generation of sequences for the content information and performing CF for the ratings, CRAE is able to significantly outperform the state of the art on both the recommendation task based on ratings and the sequence generation task based on content information.
This paper proposes a joint model of sequential content signals (such as movie plot text) and user ratings.  A stochastic recurrent neural network (RNN) is used to model the former, then producing a latent representation v that serves as the item component in the factorization of the ratings matrix.  Several auxiliary techniques are proposed to improve performance: the aforementioned noise in the RNN, a dropout-like technique called 'wildcard denoising', and a Beta pooling operation.  Experiments against five baseline models are performed on the CiteULike and Netflix datasets for recommendation and sequence generation tasks.  The CRAE shows superior to competitive recall at k and superior BLEU scores.          
Making recommendations by exploiting the plethora of text data that accompanies products seems like an under-explored area, especially using recent advances in deep language models.  I commend the authors for contributing to this research direction.  The CRAE seems to work well (at least in recall at k), performing at a high level on two real-world datasets.  However, I think this paper would be a better fit for a more applied conference, such as KDD or RecSys, because there is little novelty to the model's core components.  I'll address each individually, in order of (my perceived) importance:  1) Robust Recurrent Networks (RRN): The proposed RRN uses distributional activations that are backpropagated through directly.  If this development was indeed as novel as the paper seems to suggest, then my rating would have been higher (probably "poster level" in the 'novelty' category).  However, these exact techniques are used in "Fast Dropout Training" by S. Wang and C. Manning: http://nlp.stanford.edu/pubs/sidaw13fast.pdf  This paper is not cited but absolutely should be.  The RRN is simply a recurrent version of the Wang-Manning model.  Furthermore, considering the RNN from Wang & Manning's asymptotic dropout perspective, I'm not so sure the hidden activations should be drawn from normal distributions since traditional feedforward dropout does not work well in RNNs.  Alternatives preserving activations over time seem to work better (see https://arxiv.org/abs/1606.01305 and http://arxiv.org/abs/1512.05287).  2) "CRAE as a Hierarchical Bayesian Model": Despite the RRN's lack of novelty, Bayesian learning of a model as complex as the CRAE on datasets as large as Netflix would have been a laudable contribution.  However, only MAP estimation is performed, thus relegating the priors to regularization terms.  It's unclear how Bayesian learning of this model could be done at any realistic scale (which the authors admit, line 178) and so I don't think it's accurate to call it a Bayesian model (for example, one wouldn't call a neural network with L2 regularization a Bayesian neural network).  3) Wildcard denoising: Sampling a 'wildcard' word does seem to be a novel regularization technique that others may find useful.  Although, the development itself is not novel enough to significantly increase the paper's score.    4) Beta pooling:  The authors suggest pooling latent representations via partitioning the Beta distribution and weighting each vector by the corresponding area under the pdf.  This would be interesting if the Beta parameters were learned (which they can be, as the supplementary materials discuss), but they seem to be set a priori in the experiments (from line 186).  When the hyperparameters are fixed, the method reduces to weighted average pooling (I think?).  This is not significantly new enough to warrant an increase from 'sub-standard'.    Next, I'll move on to address the experiments.  The authors test the CRAE for making recommendations using recall at K, saying precision at K "is not suitable" "due to the fact that [a zero entry may denote] the user is not interested in the item, or that the user is not aware of its existence."  While I agree with this statement, the Netflix dataset has ratings--why not report precision at k for that data?