Q-LDA: Uncovering Latent Patterns in Text-based Sequential Decision Processes
In sequential decision making, it is often important and useful for end users to understand the underlying patterns or causes that lead to the corresponding decisions. However, typical deep reinforcement learning algorithms seldom provide such information due to their black-box nature. In this paper, we present a probabilistic model, Q-LDA, to uncover latent patterns in text-based sequential decision processes. The model can be understood as a variant of latent topic models that are tailored to maximize total rewards; we further draw an interesting connection between an approximate maximum-likelihood estimation of Q-LDA and the celebrated Q-learning algorithm. We demonstrate in the text-game domain that our proposed method not only provides a viable mechanism to uncover latent patterns in decision processes, but also obtains state-of-the-art rewards in these games.
This paper targets on two text games and propose a new reinforcement learning framework Q-LDA to discover latent patterns in sequential decision process. The proposed model uses LDA to convert action space into a continuous representation and subsequently use Q-learning algorithm to iteratively make decision in a sequential manner.
      
Authors apply the proposed model to two different text games, and achieve better performance than previous proposed baseline models.
      
The paper is a little bit hard to follow with some missing or inconsistent information. The paper is not self-contained, for a reader that is not familiar with the problem domain, one may need to refer to the Appendix or prior works almost all the time.
Some detailed comments:
      
- I would suggest authors to include a detailed section highlighting the contribution of the paper.
- Authors provide some screenshots on the text game interface in the appendix material, but the information of the text games is still short. The reference [11] also doesn't provide much useful context neither. I would recommend authors to include some example text flow (at least in the Appendix) from these games to better illustrate the target scenario. What are and how many is the possible conversation flow of each text game? 
      
- In the context of the game, the agent only receives a reward at the end of game. This is consistent with the text in line 74-75. However, in the graphically model shown in Figure 1. It seems like there is a reward after each turn. I assume this graphical illustration is for general process, but it should be nice to include an explanation in the text.
      
- In the graphically illustration, it is unclear to me which variables are observable and which are not. For example, all \beta_A and \beta_S are not observable. The rewards r_t's are not observable in my understanding.
      
- In the generative process, it occurs to me that the observation text W are generated following LDA process as well, but in the graphically model illustration, there is a missing plate in the figure.
      
- I would encourage authors to include a section of model complexity analysis, for example, what is the complexity of parameter space. Given the size of the dataset and the complexity of the proposed model, it is hard to judge if the learned policy is generalizable.
      
- In the experiments, what are the vocabulary size of the dataset, in terms of observed text and action text? In Table 2 and Figure 2, authors demonstrate some interesting outcome and observation from the generated topics. I am wondering are these topics being used in other episode, since these topics look very fine-grained and hence may not be applicable to other scenario.
      
- I would also recommend to add in a list of all topics (maybe in Appendix), rather than the cherry-picked ones, to display.
      
- In the experiments, authors mention that the upper bound for reward is 20 for "Saving John" and 30 for "Machine of Death". Are these reward numbers objectively specified in the game flow or assigned in a post-hoc manner? If latter, how do you justify this measure (especially if the reward are assigned upon game termination).
      
- Missing reference in line 76. Missing space in line 106. {D}irichlet in line 291.
%%%%%%%%%%%%%%%%%%%%%%%%
The authors' response have clarified some of my questions and they also agree to improve the paper to make it more self contained. I have adjusted the score accordingly.