Answers.Body: <p>I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).</p>
  <p>Q learning is an <em>off policy</em> algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, however, before the Q table has converged, we want the agent to explore the environment to collect the data it needs (without sacrificing too much return, this is the <em>exploration-exploitation tradeoff</em>).</p>
  <p>It is very common practice to use the intermediate Q tables to induce a policy for the agent. For example, the epsilon-greedy policy will select the best action according to the current Q table with high probability, and will otherwise choose a random action.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-02T11:35:50.840Z
Answers.LastActivityDate: 2021-07-02T11:35:50.840Z
Answers.UserDisplayName: harwiltz
Body: <p>Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-table for given state i.e., the action with maximum Q-value. So, in case of second player, a Q-table is required.</p>
  <p>It is known that <strong>Q-table has to be constructed only by running several episodes</strong> after its arbitrary initialization. So, the two players has to play with some policy to construct a Q-table for player two. Since the first player uses random policy. I have doubt regarding the policy of second player for constructing Q-table.</p>
  <p>I have doubt regarding the policy of second player only. I know the policy he need to follow after the completion of updation of Q-table. I am not sure about the policy he need to follow while updating the Q-table only.</p>
  <p>Which policy does my second player need to follow <strong>while constructing</strong> the Q-table for himself? Can he use random policy like player one? Or does he need to use arbitrarily initialized or partially updated Q-table itself for selecting best action? Or can he use some other policy till the completion of updating Q-table?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-06T10:45:03.063Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127228/discussion-on-question-by-satya-which-policy-has-to-be-followed-by-a-player-whil).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><off-policy-methods><exploration-exploitation-tradeoff><exploration-strategies>
Title: Which policy has to be followed by a player while construction of its own Q-table?
UserDisplayName: satya
ThreadText: # Which policy has to be followed by a player while construction of its own Q-table?
  
  (Asked by: username_0 on 2021-07-02)
  
  Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-table for given state i.e., the action with maximum Q-value. So, in case of second player, a Q-table is required.
  
  It is known that **Q-table has to be constructed only by running several episodes** after its arbitrary initialization. So, the two players has to play with some policy to construct a Q-table for player two. Since the first player uses random policy. I have doubt regarding the policy of second player for constructing Q-table.
  
  I have doubt regarding the policy of second player only. I know the policy he need to follow after the completion of updation of Q-table. I am not sure about the policy he need to follow while updating the Q-table only.
  
  Which policy does my second player need to follow **while constructing** the Q-table for himself? Can he use random policy like player one? Or does he need to use arbitrarily initialized or partially updated Q-table itself for selecting best action? Or can he use some other policy till the completion of updating Q-table?
  
  **Tags:** reinforcement-learning, q-learning, off-policy-methods, exploration-exploitation-tradeoff, exploration-strategies
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127228/discussion-on-question-by-satya-which-policy-has-to-be-followed-by-a-player-whil).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-02. Score: 3 | Accepted answer
  
  I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).
  
  Q learning is an *off policy* algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, however, before the Q table has converged, we want the agent to explore the environment to collect the data it needs (without sacrificing too much return, this is the *exploration-exploitation tradeoff*).
  
  It is very common practice to use the intermediate Q tables to induce a policy for the agent. For example, the epsilon-greedy policy will select the best action according to the current Q table with high probability, and will otherwise choose a random action.
  
  
  ---
  
  