Answers.Body: <p>I suggest you to start reading this one on <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">BERT</a> and this one on <a href="https://github.com/openai/gpt-2" rel="nofollow noreferrer">GPT-2</a>.</p>
  
  <blockquote>
    <p>I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.</p>
  </blockquote>
  
  <p>I'm not sure you want to create your chatbot using RL architecture at all. But if you want to implement such an idea the right way to go with that is by using iterative approach and starting from a really simple base:</p>
  
  <ol>
  <li>Define a list of actions for your agent. Might be 10-words
  vocabulary with simple one-word answers. </li>
  <li>Define you simple environment. It might have 20 different states (e.g. greetings/conversation starters <code>hey</code>, <code>hello</code>)</li>
  <li>Map good responses (<code>hello</code> -> <code>hi</code>) to positive rewards and <code>-1</code> for the rest of them.</li>
  <li>If you want to move forward with DQN try a simple Q-learning algorithm on your simple environment and train agent to properly response to those.</li>
  <li>Once you have done the above you can make your vocabulary bigger</li>
  <li>Now replace your table of input words which gives you a state for your network with NN/word2vec/any other NLP approach which will convert your input sentence to a state.</li>
  <li>Manually provide a reward as a response for what you agent did according to learned policy.</li>
  <li>Do that infinite amount of time and your agent probably will figure out how to respond properly.</li>
  </ol>
  
  <p><strong>NOTE</strong>: DQN uses discrete action space. So in such a case you will have only limited or REALLY HUGE actions space. It's all possible combinations between all possible words in you vocabulary (assuming <em>action</em> maps to some <em>sentence</em>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T13:01:49.223Z
Answers.Text: Thank you for a proper explanation. This definitely makes things clearer. Do you know of any implementations of such an approach in which directly the user trains the model?
Answers.UserDisplayName: mayank agrawal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T17:12:11.203Z
Answers.Text: @mayankagrawal I guess you need to at least be familiar with some base approaches, so https://www.nltk.org/book/ is a good place to start. If you want to go right away to implementing - use https://www.tensorflow.org/tutorials/sequences/text_generation or https://www.tensorflow.org/tutorials/keras/basic_text_classification to understand how text is represented with a model.
Answers.UserDisplayName: Most Wanted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-27T18:14:25.177Z
Answers.LastActivityDate: 2019-03-27T18:14:25.177Z
Answers.UserDisplayName: Most Wanted
Body: <p>I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of the conversation.</p>
  
  <p>All implementations I see for RL are trained from an environment taken from <a href="https://gym.openai.com/" rel="nofollow noreferrer">Gym</a> or created manually. These environments provide the next state, rewards etc to the model based on which it is trained.</p>
  
  <p>Since I am creating a dialogue policy model which will be trained through real user conversations, I cannot provide a "pre-defined" environment which can provide the states and rewards. I am planning to train it myself by talking to it and providing rewards and next state (which I think is called interactive learning).</p>
  
  <p>But I was not able to find any implementations, tutorials or articles on interactive learning. I am not able to figure out how to create such a model, how to take care of the episodes, sessions etc. This will be a continuous learning that will go on for months maybe. I have to save the model each day and continue training the next day by loading the model from that same state.</p>
  
  <p>Can anyone guide me in the correct direction on how to approach this? Any githubs links, articles, tutorials of such implementations will be highly appreciated. I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.</p>
  
Tags: <reinforcement-learning><q-learning><chat-bots><dqn>
Title: How to build a DQN agent which can be trained through interactive learning?
UserDisplayName: mayank agrawal
ThreadText: # How to build a DQN agent which can be trained through interactive learning?
  
  (Asked by: username_0 on 2019-03-27)
  
  I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of the conversation.
  
  All implementations I see for RL are trained from an environment taken from [Gym](https://gym.openai.com/) or created manually. These environments provide the next state, rewards etc to the model based on which it is trained.
  
  Since I am creating a dialogue policy model which will be trained through real user conversations, I cannot provide a "pre-defined" environment which can provide the states and rewards. I am planning to train it myself by talking to it and providing rewards and next state (which I think is called interactive learning).
  
  But I was not able to find any implementations, tutorials or articles on interactive learning. I am not able to figure out how to create such a model, how to take care of the episodes, sessions etc. This will be a continuous learning that will go on for months maybe. I have to save the model each day and continue training the next day by loading the model from that same state.
  
  Can anyone guide me in the correct direction on how to approach this? Any githubs links, articles, tutorials of such implementations will be highly appreciated. I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.
  
  **Tags:** reinforcement-learning, q-learning, chat-bots, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-27. Score: 1
  
  I suggest you to start reading this one on [BERT](https://github.com/google-research/bert) and this one on [GPT-2](https://github.com/openai/gpt-2).
  
  > I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.
  
  I'm not sure you want to create your chatbot using RL architecture at all. But if you want to implement such an idea the right way to go with that is by using iterative approach and starting from a really simple base:
  
  1. Define a list of actions for your agent. Might be 10-words
     vocabulary with simple one-word answers.
  2. Define you simple environment. It might have 20 different states (e.g. greetings/conversation starters `hey`, `hello`)
  3. Map good responses (`hello` -> `hi`) to positive rewards and `-1` for the rest of them.
  4. If you want to move forward with DQN try a simple Q-learning algorithm on your simple environment and train agent to properly response to those.
  5. Once you have done the above you can make your vocabulary bigger
  6. Now replace your table of input words which gives you a state for your network with NN/word2vec/any other NLP approach which will convert your input sentence to a state.
  7. Manually provide a reward as a response for what you agent did according to learned policy.
  8. Do that infinite amount of time and your agent probably will figure out how to respond properly.
  
  **NOTE**: DQN uses discrete action space. So in such a case you will have only limited or REALLY HUGE actions space. It's all possible combinations between all possible words in you vocabulary (assuming *action* maps to some *sentence*)
  
  > **Comments:**
  > **username_0**: Thank you for a proper explanation. This definitely makes things clearer. Do you know of any implementations of such an approach in which directly the user trains the model?  
  > **username_1**: @mayankagrawal I guess you need to at least be familiar with some base approaches, so https://www.nltk.org/book/ is a good place to start. If you want to go right away to implementing - use https://www.tensorflow.org/tutorials/sequences/text_generation or https://www.tensorflow.org/tutorials/keras/basic_text_classification to understand how text is represented with a model.  
  
  ---
  
  