Body: <p>This is my first question on this forum and I would like to welcome everyone.
  I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems to be properly initialized does not learn against random opponent. Actually the score is about 50-60% won games out of nearly 500. Generally if it gets some score after first 20-50 episodes it stays on the same level. I have doubts on the process of learning and how to decide when the agent is trained. Current flow is as follows:</p>
  <ol>
  <li>Initialize game state.</li>
  <li>With epsilon greedy policy choose the action to make based on currently available actions depending on game state</li>
  <li>Get opponent to make his action</li>
  <li>Get the reward as number of flipped places that remain after opponent move.</li>
  <li>Save the observation to replay buffer</li>
  <li>If number of elements in replay buffer equals or higher than batch size do the training.</li>
  </ol>
  <p>What I do not know is when do I know when to stop the training. Previously this agent trained against MinMax algorithm learned how to win 100% games because MinMax played exactly the same every time. I would like the agent to generalize the game. Right now I save the network weights after the game is won but I think it does not matter. I can't see that this agent find some policy and improve over time. Whole code for the environment, agent and training loop can be found here: <a href="https://github.com/MikolajMichalski/RL_othello_mgr" rel="nofollow noreferrer">https://github.com/MikolajMichalski/RL_othello_mgr</a> I would appreciate any help. I would like to understand how the RL works :)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-23T16:07:48.227Z
Comments.Text: Hi Mikołaj and welcome! The best strategy for the game of reversi is somewhat counterintuitive. It's actually preferable to have little pieces of your color on the board at the beginning or midgame, and only in the very end flip the situation in your favor. The way you currently define your reward function encourages the agent to be greedy in the beginning, which may lead to losing the control of the corners (which are important). You want the agent to learn to win by itself, and your reward function should only award points for winning/drawing and losing.
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-28T15:57:23.683Z
Comments.Text: Hi, thank you for the answer. I changed reward to be only given after the episode is finished. Now it's counted as learning_agent_score - opponent_score. I can see a bit of improvement but is is not satisfying enough. I may experiment with hyper parameters. I have other question... Is it a proper way to train agent against random opponent? I thought about the policy of setting opponents weights to learning agent weights every time the win percentage gets higher so the trained agent plays against better opponent over time.
Comments.UserDisplayName: Mikołaj Michalski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-28T17:06:24.693Z
Comments.Text: To best of my knowledge, self-play would preferred over playing versus random opponent in terms of learning efficiency, since the state space is huge. Training network against itself using some minmax variation should be able to pick up some strategies. You can then evaluate the performance of the network against random opponent. Though, it might take a good while before agent learns anything the game is very complex (less than, but comparable to chess).
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-10T13:57:46.250Z
Comments.Text: As I did some experiments with hyperparameters and did not see much progress... I am coming back with more questions :) 
  1. What would be the best network architecture in this case? Mine is 1st layer - input 64 of size with ReLU, 2nd layer - hidden with ReLU- 32 of size, 3rd layer - 64 of size linear and 4th with softmax. I am not sure if this is proper approach but I can't find good source about building good network arch. 
  2. Currently I am getting minibatches with random samples for training. Is it good as the states sequentions for each games are  highly dependent?
Comments.UserDisplayName: Mikołaj Michalski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-10T14:05:01.150Z
Comments.Text: And another one:
  3. Should the training be done after each episode or each step?
Comments.UserDisplayName: Mikołaj Michalski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T14:42:18.663Z
Comments.Text: Hello again! I worked on a similar problem before - I tried to solve mancala game with DDQN (which I could not solve with that algorithm and currently trying some others). For the network architecture I only found vague suggestion -- seems that this topic is not well understood within the community. I'm inclined to think that you may need more neurons in each layer (a couple of hundreds), because neurons need to look at relations between pieces too and that's a huge space.
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T14:42:22.377Z
Comments.Text: The second question - yes, that's fine. DDQN should converge to optimal policy in theory, regardless of what you're feeding to it.
  
  The third - the training should be done at the end of the episode, provided, there's enough samples.
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T14:45:59.287Z
Comments.Text: I would also suggest to test your algorithm on a simpler game for a sanity check. Maybe not as simple as tic-tac-toe, but something that  has smaller state/action space and hence is easier for agent to find patterns in
Comments.UserDisplayName: mark mark
Tags: <reinforcement-learning><training><dqn><deep-rl><double-dqn>
Title: DDQN Agent in Othello (Reversi) game struggle to learn
UserDisplayName: Mikołaj Michalski
ThreadText: # DDQN Agent in Othello (Reversi) game struggle to learn
  
  (Asked by: username_0 on 2020-12-23)
  
  This is my first question on this forum and I would like to welcome everyone.
  I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems to be properly initialized does not learn against random opponent. Actually the score is about 50-60% won games out of nearly 500. Generally if it gets some score after first 20-50 episodes it stays on the same level. I have doubts on the process of learning and how to decide when the agent is trained. Current flow is as follows:
  
  1. Initialize game state.
  2. With epsilon greedy policy choose the action to make based on currently available actions depending on game state
  3. Get opponent to make his action
  4. Get the reward as number of flipped places that remain after opponent move.
  5. Save the observation to replay buffer
  6. If number of elements in replay buffer equals or higher than batch size do the training.
  
  What I do not know is when do I know when to stop the training. Previously this agent trained against MinMax algorithm learned how to win 100% games because MinMax played exactly the same every time. I would like the agent to generalize the game. Right now I save the network weights after the game is won but I think it does not matter. I can't see that this agent find some policy and improve over time. Whole code for the environment, agent and training loop can be found here: <https://github.com/MikolajMichalski/RL_othello_mgr> I would appreciate any help. I would like to understand how the RL works :)
  
  **Tags:** reinforcement-learning, training, dqn, deep-rl, double-dqn
  > **Comments:**
  > **username_1**: Hi Mikołaj and welcome! The best strategy for the game of reversi is somewhat counterintuitive. It's actually preferable to have little pieces of your color on the board at the beginning or midgame, and only in the very end flip the situation in your favor. The way you currently define your reward function encourages the agent to be greedy in the beginning, which may lead to losing the control of the corners (which are important). You want the agent to learn to win by itself, and your reward function should only award points for winning/drawing and losing.  
  > **username_0**: Hi, thank you for the answer. I changed reward to be only given after the episode is finished. Now it's counted as learning_agent_score - opponent_score. I can see a bit of improvement but is is not satisfying enough. I may experiment with hyper parameters. I have other question... Is it a proper way to train agent against random opponent? I thought about the policy of setting opponents weights to learning agent weights every time the win percentage gets higher so the trained agent plays against better opponent over time.  
  > **username_1**: To best of my knowledge, self-play would preferred over playing versus random opponent in terms of learning efficiency, since the state space is huge. Training network against itself using some minmax variation should be able to pick up some strategies. You can then evaluate the performance of the network against random opponent. Though, it might take a good while before agent learns anything the game is very complex (less than, but comparable to chess).  
  > **username_0**: As I did some experiments with hyperparameters and did not see much progress... I am coming back with more questions :)  1. What would be the best network architecture in this case? Mine is 1st layer - input 64 of size with ReLU, 2nd layer - hidden with ReLU- 32 of size, 3rd layer - 64 of size linear and 4th with softmax. I am not sure if this is proper approach but I can't find good source about building good network arch.  2. Currently I am getting minibatches with random samples for training. Is it good as the states sequentions for each games are  highly dependent?  
  > **username_0**: And another one: 3. Should the training be done after each episode or each step?  
  > **username_1**: Hello again! I worked on a similar problem before - I tried to solve mancala game with DDQN (which I could not solve with that algorithm and currently trying some others). For the network architecture I only found vague suggestion -- seems that this topic is not well understood within the community. I'm inclined to think that you may need more neurons in each layer (a couple of hundreds), because neurons need to look at relations between pieces too and that's a huge space.  
  > **username_1**: The second question - yes, that's fine. DDQN should converge to optimal policy in theory, regardless of what you're feeding to it.  The third - the training should be done at the end of the episode, provided, there's enough samples.  
  > **username_1**: I would also suggest to test your algorithm on a simpler game for a sanity check. Maybe not as simple as tic-tac-toe, but something that  has smaller state/action space and hence is easier for agent to find patterns in  
  
  ---
  
  ## 0 answers
  
  