Body: <p>I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great results for Connect 4. The agent is learning to win quickly, if it gets the chance, but it only plays in the centre. It is unable to detect threats in the first and last columns. I am using DDQN with memory replay. Also <code>teacher</code> and <code>student</code> refer to two agents at different strengths, while the teacher is frequently replaced by a new student. My algorithm looks simplified as follows:</p>
  <pre><code>for i in range(episodes):
      observation = env.reset()
      done = False
      while not done:
          if env.turn == 1:
              action = student.choose_action(observation)
              observation_, reward, done, info = env.step(action)
              loss = student.learn(observation, action, reward, observation_, done))
              observation = observation_
          else:
              action = teacher.choose_action(-observation)
              observation_, reward, done, info = env.step(action)
              observation = observation_
  </code></pre>
  <p>The observation is -1 for player &quot;o&quot;, 1 for player &quot;x&quot; and 0 for empty. The agent learns to play as player &quot;x&quot; and through <code>action = teacher.choose_action(-observation)</code> it should find the best move for player &quot;o&quot;. I hope that is clear.</p>
  <p>The update rule looks as follows:</p>
  <pre><code># Get predicted q values for the actions that were taken
  q_pred = Q_eval.forward(state, action)
  # Get Q value for opponent's next move
  state_ *= -1.
  q_next = Q_target.forward(state_, max_action)
  # Update rule
  q_target = reward_batch - gamma * q_next * terminal
  loss = Q_eval.loss(q_pred, q_target)
  </code></pre>
  <p>I am using <code>-gamma * q_next * terminal</code>, because the reward is negative, if the opponent wins in the next move. Am I missing anything important or is it just a question of hyperparameter tuning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-15T08:42:56.660Z
Comments.Text: What does `terminal` represent, and what values can it take?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-15T09:08:18.977Z
Comments.Text: it is `0` if the next state is terminal (`done == True`) and `1` otherwise. This is, because all terminal states have Q-Value of zero in Q-Learning.
Comments.UserDisplayName: spadel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-17T00:25:34.690Z
Comments.Text: You also asked this question [here](https://stackoverflow.com/q/67542074/3924118).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-17T09:59:35.587Z
Comments.Text: yeah I did, and?
Comments.UserDisplayName: spadel
Tags: <reinforcement-learning><q-learning><deep-rl><dqn><double-dqn>
Title: Update Rule with Deep Q-Learning (DQN) for 2-player games
UserDisplayName: spadel
ThreadText: # Update Rule with Deep Q-Learning (DQN) for 2-player games
  
  (Asked by: username_0 on 2021-05-14)
  
  I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great results for Connect 4. The agent is learning to win quickly, if it gets the chance, but it only plays in the centre. It is unable to detect threats in the first and last columns. I am using DDQN with memory replay. Also `teacher` and `student` refer to two agents at different strengths, while the teacher is frequently replaced by a new student. My algorithm looks simplified as follows:
  
  ```
  for i in range(episodes):
      observation = env.reset()
      done = False
      while not done:
          if env.turn == 1:
              action = student.choose_action(observation)
              observation_, reward, done, info = env.step(action)
              loss = student.learn(observation, action, reward, observation_, done))
              observation = observation_
          else:
              action = teacher.choose_action(-observation)
              observation_, reward, done, info = env.step(action)
              observation = observation_
  
  ```
  
  The observation is -1 for player "o", 1 for player "x" and 0 for empty. The agent learns to play as player "x" and through `action = teacher.choose_action(-observation)` it should find the best move for player "o". I hope that is clear.
  
  The update rule looks as follows:
  
  ```
  # Get predicted q values for the actions that were taken
  q_pred = Q_eval.forward(state, action)
  # Get Q value for opponent's next move
  state_ *= -1.
  q_next = Q_target.forward(state_, max_action)
  # Update rule
  q_target = reward_batch - gamma * q_next * terminal
  loss = Q_eval.loss(q_pred, q_target)
  
  ```
  
  I am using `-gamma * q_next * terminal`, because the reward is negative, if the opponent wins in the next move. Am I missing anything important or is it just a question of hyperparameter tuning?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, dqn, double-dqn
  
  > **Comments:**
  > **username_1**: What does `terminal` represent, and what values can it take?  
  > **username_0**: it is `0` if the next state is terminal (`done == True`) and `1` otherwise. This is, because all terminal states have Q-Value of zero in Q-Learning.  
  > **username_2**: You also asked this question [here](https://stackoverflow.com/q/67542074/3924118).  
  > **username_0**: yeah I did, and?  
  
  ---
  
  ## 0 answers
  
  