Answers.Body: <p>Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
  The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
  It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.</p>
  
  <p>Epsilon is a parameter that governs the trade-off between <em>exploration</em> and <em>exploitation</em>. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (<em>exploitation</em>), or try something new (<em>exploration</em>). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.</p>
  
  <p>You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
  Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
  In the example that you posted, you can see that Pacman starts off doing nothing but <em>exploring</em>, but after 500000 games it <em>exploits</em> 95% of the time. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T15:39:02.813Z
Answers.Text: To augment the last paragraph I would add that the exploitation vs exploration mix usually needs to keep the agent *close* to optimal behaviour towards the end, even for off-policy learners (e.g. Q-learning) because the agent needs to learn about what happens close to the optimal path in order to make minor switches of action. In an environment that is challenging to complete (e.g. many steps before top rewards), then a high epsilon won't ever reach states that are interesting for optimal control.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T15:47:00.717Z
Answers.Text: @NeilSlater Yes, that is a good explanation. I'll edit to include.
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T12:41:35.310Z
Answers.LastActivityDate: 2019-05-30T15:50:02.477Z
Answers.LastEditDate: 2019-05-30T15:50:02.477Z
Answers.UserDisplayName: DrMcCleod
Body: <p>I am tentatively reusing a codebase of <a href="https://github.com/prasoon2211/pacman-RL/blob/master/pacman_vanilla_DQN/pacman_agent.py" rel="nofollow noreferrer">pacman</a> to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:</p>
  
  <ol>
  <li><p>How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the <code>replay_memory_size</code> as 400. Would that make sense? </p></li>
  <li><p>What is the return value <code>epsilon</code> when calling function <code>PiecewiseSchedule</code>? I also proportionally decrease its parameters as follows:</p></li>
  </ol>
  
  <pre><code>        epsilon = PiecewiseSchedule([(0, 1.0),
                                       (40, 1.0), # since we start training at 10000 steps
                                       (80, 0.4),
                                       (200, 0.2),
                                       (400, 0.1),
                                       (2000, 0.05)], outside_value=0.01)
          replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
  </code></pre>
  
  <p>where the original function call is like this:</p>
  
  <pre><code>        epsilon = PiecewiseSchedule([(0, 1.0),
                                       (10000, 1.0), # since we start training at 10000 steps
                                       (20000, 0.4),
                                       (50000, 0.2),
                                       (100000, 0.1),
                                       (500000, 0.05)], outside_value=0.01)
          replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
  </code></pre>
  
  <p>And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function <code>PiecewiseSchedule</code>? Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-30T10:10:17.400Z
Comments.Text: Just to correct you, it is not reply memory, it is replay memory
Comments.UserDisplayName: Kartik Podugu
Tags: <deep-learning><reinforcement-learning>
Title: Understanding the configuration of replay memory and epsilon in deep reinforcement learning
UserDisplayName: lllllllllllll
ThreadText: # Understanding the configuration of replay memory and epsilon in deep reinforcement learning
  
  (Asked by: username_0 on 2019-05-30)
  
  I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:
  
  How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the replay_memory_size as 400. Would that make sense? 
  What is the return value epsilon when calling function PiecewiseSchedule? I also proportionally decrease its parameters as follows:
  
          epsilon = PiecewiseSchedule([(0, 1.0),
                                       (40, 1.0), # since we start training at 10000 steps
                                       (80, 0.4),
                                       (200, 0.2),
                                       (400, 0.1),
                                       (2000, 0.05)], outside_value=0.01)
          replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
  
  where the original function call is like this:
          epsilon = PiecewiseSchedule([(0, 1.0),
                                       (10000, 1.0), # since we start training at 10000 steps
                                       (20000, 0.4),
                                       (50000, 0.2),
                                       (100000, 0.1),
                                       (500000, 0.05)], outside_value=0.01)
          replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
  
  And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function PiecewiseSchedule? Thank you!
  
  **Tags:** deep-learning, reinforcement-learning
  > **Comments:**
  > **username_1**: Just to correct you, it is not reply memory, it is replay memory  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-05-30. Score: 4 | Accepted answer
  
  Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
  The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
  It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.
  
  Epsilon is a parameter that governs the trade-off between *exploration* and *exploitation*. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (*exploitation*), or try something new (*exploration*). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.
  
  You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
  Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
  In the example that you posted, you can see that Pacman starts off doing nothing but *exploring*, but after 500000 games it *exploits* 95% of the time.
  
  > **Comments:**
  > **username_3**: To augment the last paragraph I would add that the exploitation vs exploration mix usually needs to keep the agent *close* to optimal behaviour towards the end, even for off-policy learners (e.g. Q-learning) because the agent needs to learn about what happens close to the optimal path in order to make minor switches of action. In an environment that is challenging to complete (e.g. many steps before top rewards), then a high epsilon won't ever reach states that are interesting for optimal control.  
  > **username_2**: @NeilSlater Yes, that is a good explanation. I'll edit to include.  
  
  ---
  
  