We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. We demonstrate the actor-critic's ability to significantly improve the performance of a variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.
The work presents a two-level architecture for building an efficient ranking solution for standard collaborative filtering task in recommender systems. The core of this solution is the actor-critic approach, where the critic (motivated by ideas from RL) tries to directly approximate a ranking-based metric using as an input prediction from the actor (VAE framework) and the ground truth knowledge from an Oracle. Both actor and critic require a pre-training step, after which the critic is set to propagate adjustments to actor parameters. Experiments demonstrate the advantage of such approach over competing methods.
The work presents an interesting idea of feature-based critic for learning a better ranking. The choice of features is surprisingly effective considering how simple they are – just basic statistics. On the other hand, the motivating example with NLL vs NDCG is not convincing. Comparing to much simpler linear SOTA methods, the improvement over multinomial VAE is not impressive. This is probably due to VAE being not a real SOTA in the first place, however other possible candidates for actor were not demonstrated. Anyway, I believe that this is an important direction for research and can be considered for acceptance if the authors adjust the text according to the comments below. 
Main arguments
The text is well structured, easy to read. Key concepts are explained carefully with the appropriate level of details. Open-source repository with the source code is definitely a plus. However, there’s a number of inaccuracies, which should be addressed.
In the example demonstrating VAE's pitfalls, it is stated that prediction A is intuitively better than prediction B. I’d argue with that intuition. For me, B looks more consistent as there’s a smaller gap between predictions for positive targets. In A one of the predictions is much lower, whcih seems more like an outlier. Moreover, shouldn’t NDCG be 1 in both cases? The ordering of positive examples is the same in both cases. Please, consider addiing more explanations to clarify your intuition or replace the example with something less contradicting. 
The  statement “making RaCT scalable to large-scale datasets” is arguable from the technical viewpoint. As far as I know, there’s still no good support for sparse calculations in NN frameworks. When you take user mini batch you have to convert it to a dense matrix with explicitly stored zeros. It’s not a problem for the datasets tested in the paper (with <100K items). However, consider 100M items. What will be the size of user batch that will fit GPU memory? Will it be still large enough and effective from the learning viewpoint (smaller batches are likely to reduce generalization)? As it will mostly contain zeroes (few know interactions vs ~100M unknown), efficiency of calculations with this batch will also be low comparing to operations in sparse format. In that sense the datasets used in the paper are not really large scale.
Another false statement: “This is much more efﬁcient than most traditional latent factor collaborative ﬁltering models …, where a time-consuming optimization procedure is typically performed to obtain the latent factor for a user who is not present in the training data”. All factorization models admit folding-in procedure, which doesn’t depend on the size of the problem (i.e., the number of users or items) and depends only on the rank of decomposition (the number of latent variables). This is among the reasons why MF is so popular in industrial applications.
Furthermore, SVD-based models (I’m talking about the real SVD, not MF models like SVD++) are absolute winners here as they do not even require running any additional optimization steps. Due to the orthogonality property, the solution for folding-in has an analytical form in the case of SVD, e.g., given a low-rank item embeddings matrix V with orthonormal columns, the folding-in solution (vector of predicted item scores for a user) is simply given by an orthogonal projector
r = VV^T p,
for any vector of user preferences p, be it a known user or a new one (i.e., not present in the training data). It is basically a simple encoder-decoder architecture with a single linear layer. Training SVD-based models can also very efficient and can be scaled to almost billion-size problems, e.g. https://github.com/criteo/Spark-RSVD. Moreover, one can significantly improve standard PureSVD model [Cremonesi, Koren, Turrin 2010] by a simple scaling trick as it is shown in the EigenRec model [Nikolakopoulos 2019], you can also see more details on this in the blog https://www.eigentheories.com/blog/to-svd-or-not-to-svd. I also find it interesting that the scaling trick in SVD partially intersects with the idea of importance of user behavior statistics in the critic model,  as it also depends on how many clicks a user has made.
I ran my own experiments with VAE and SVD in google colab (I can provide a link to the notebook if needed). With a proper tuning the SVD-based solution gives NDCG=0.415, which is within 3% difference of the Mult-VAE’s best result on the ML-20M dataset. It takes around 200s to train the SVD model of rank 600 on the CPU in the google colab. For comparison, one epoch of VAE training takes 26s in the same google colab environment (which I believe is using Tesla K80 GPU). The same NDCG level is achieved by Mult-VAE after approximately 20 epochs, which amounts to more than 500s of training time, twice longer than SVD. Speaking about practicality, even a student will have a decent CPU nowadays, but not everyone will have Tesla K80. SVD merely requires doing `from scipy.sparse.linalg import svds` in most cases. Moreover, there are even better solutions than SVD (in terms of accuracy), still linear, not even based on MF, e.g., RecWalk [Nikolakopoulos and Karypis 2019] and even better one - Personalized Diffusions model [Nikolakopoulos, Berberidis, Karypis, Giannakis 2019] - that are shown to significantly outperform VAE models.
The general conclusion is that even though you improve VAE, you are actually starting from not the best model and there’s much more room for improvement (demonstrated by much simpler models) than you show in your work. Please consider this fact and adjust wording in your text accordingly. Especially in the statements like “VAEs signiﬁcantly outperform many state-of-the-art methods” or “the proposed RaCT also signiﬁcantly outperforms other state-of-the-art methods, including VAE, CDAE, WMF and SLIM” as it is really about outperforming strong baselines not real SOTA.
In addition to that, recent studies show that many modern neural networks-based approaches underperform even simple KNN-based models, see work by [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”. It includes VAE, DAE and NeuCF that you reference in your paper. 
In spite of that, the statement “[LVM]… may yield suboptimal performance, especially for large datasets (He et al., 2017)” is at best questionable and requires more investigation as the referenced here paper is about NeuCF. This also makes me doubt in the claims, where you talk about approximation of the real ranking measure by the critic. Examples with RecWalk and PersDiff show that even without optimizing any ranking metric you can get a decent quality in terms of NDCG. Even though I find the general explanation in the text intuitive and appealing, I believe, more rigorous research should be done before we can have something reliable to say about ranking metric approximation. I’d suggest being more careful with such statement in the text. Even in your own experiments a simple EASE model is sometimes better, even though it has nothing to do with approximating any elaborate ranking metric (it's closer to the nearest neighbours search).
Other comments
1)	“performance comes at a computational cost, as inference requires multiplication with an unfactored M ×M matrix” is arguable to a certain extent. In the case of SLIM there’s a sparse matrix with zero diagonal, so once it’s computed and stored in a sparse format, it’s not really a problem to do calculations with it (the training is expensive, though).
2)	 “which is argued to be a close proxy to ranking loss” a reference required here.
3)	The two-level architectures are very popular in general. At the first level, a compact list of candidates is generated by a quick to compute algorithm like MF,  then the second-level algorithm based on decision-trees or multiarmed bandits (MAB) is trained on a much smaller output space. Comparing with such approaches (especially with MAB as they are widely used in industry, probably the second popular approach after MF) would make the work more complete.
4)	“Previous work on learning-to-rank has been explored this question…” something is not right here.
5)	Unfortunately, a lot of prior studies do not report confidence intervals, and this paper is not an exception. It makes it hard to reason about statistical significance of the results and about generalization capabilities of the approach.
References:
Cremonesi, Paolo, Yehuda Koren, and Roberto Turrin. "Performance of recommender algorithms on top-n recommendation tasks." In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46. ACM, 2010.
Nikolakopoulos, Athanasios N., Vassilis Kalantzis, Efstratios Gallopoulos, and John D. Garofalakis. "EigenRec: generalizing PureSVD for effective and efficient top-N recommendations." Knowledge and Information Systems 58, no. 1 (2019): 59-81.
Nikolakopoulos, Athanasios N., and George Karypis. "Recwalk: Nearly uncoupled random walks for top-n recommendation." In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 150-158. ACM, 2019.
Nikolakopoulos, Athanasios N., Dimitris Berberidis, George Karypis, and Georgios B. Giannakis. "Personalized diffusions for top-n recommendation." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 260-268. ACM, 2019.
Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. "Are we really making much progress? A worrying analysis of recent neural recommendation approaches." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.