There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging.
In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.
This paper proposes to frame the recommendation problem as one of (model-based) RL. The two main innovations are: 1) a model and objective for learning the environment and reward models; 2) a cascaded DQN framework for reasoning about a combinatorial number of actions (i.e., which subset of items to recommend to the user).
The problem is clearly important and the authors' approach focuses on solving some of the current issue with deployment of RL-based recommenders. Overall the paper is relatively easy to follow, but the current version is not the easiest to understand and, in particular, it may be worth providing more intuitions (e.g., about the GAN-like setup). I also found that several decisions are not properly justified. The novelty of this paper seems reasonably high but my impression is that other/stronger baselines would make the study more convincing. Copy-editing the paper would also greatly improve readability.
Detailed comments:
- I am not clear on whether or not in the proposed model, users are "allowed" to not click on a recommendation. It sounds like the authors in fact allow it but I think that could be made clearer.
- Section 4. I am not sure that using the Generative Adversarial Network terminology is useful here. Specifically, it is not clear what is your generative model over (I imagine next state and reward?).
  
- Remark in Section 4.1: It seems like a user not clicking on something is also useful information. Why not model it?
  
- I am a bit unclear on the value of Lemma 1. Further, what are the assumptions behind it? (also what is this temperature parameter eta?)
  
- In Section 4.2, the size of your model seems to grow linearly with the number of user interactions. That seems like a major advantage of RNNs/LSTMs. In practice, I imagine you cull the history at some fixed point?
  
- What is the advantage of learning a reward? E.g., a very simple reward would be to give a positive reward if a user clicks on a recommended item and a negative reward otherwise. What does your learned reward allow beyond this?
  
- Section 4.3. I also found Section 4.3 to be relatively unclear. I find that more intuition would be helpful.
  Also, if Eq. 7 is equivalent to Eq. 8, then why is the solution of 8 used only to initialize 7? I guess it may have to do with not finding the global optimum.
- Your cascading DQN idea seems like a good one. It would be nice to check if the constraints are correctly learned. If not, this seems like it would do not better than a greedy action-by-action solution. Is that correct?
  
- In Section 6.1, it would be good to discuss the pre-processing in the main text since it's pretty important to understand the study (e.g., evaluate is impact).
  
- In 6.2, your baselines seem a bit weak. Why not compare to more recent CF models (e.g., including Session-Based RNNs which you cite earlier)?
  
- Related work: it would probably be good to survey some of the multi-arm bandit literature. There is also some CF-RL work which should be cited (perhaps there are a few things in there that should be compared to in Section 6.3 & 6.4). 
- Section 6.2 and Table 1. I believe that Recall@k is most common in recommendation-systems-for-implicit-data literature. Or, are you assuming that what people do not click on are true negatives? This doesn't seem quite right as users are only allowed to click on a single item.
- In Section 6.3, could you clarify how do you learn your reward model that is used to train the various methods?
- There are many typos and grammatical errors in the paper. I would suggest that the authors carefully copy-edit the manuscript.