Answers.Body: <p>I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and <strong>experience replay</strong>), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN.</p>
  <p>Let's take a look at the <code>observation</code> method.</p>
  <pre><code>def observation(self, observation):
      self.buffer[:-1] = self.buffer[1:]
      self.buffer[-1] = observation
      return self.buffer
  </code></pre>
  <ul>
  <li><p><code>self.buffer[:-1] = self.buffer[1:]</code> essentially drops/forgets the first (which is also the oldest) observation in the buffer.</p>
  </li>
  <li><p><code>self.buffer[-1] = observation</code> adds the new <code>observation</code> (passed as parameter) as the last element of the buffer</p>
  </li>
  </ul>
  <p>You can execute the following code to see what that method does.</p>
  <pre><code>buffer = [10, 7, 3]
  print(buffer)
  
  buffer[:-1] = buffer[1:]
  
  observation = 5
  buffer[-1] = observation
  
  print(buffer) # [7, 3, 5]
  </code></pre>
  <p>If you are not familiar with the <strong>experience replay</strong> technique, you can take a look at <a href="https://ai.stackexchange.com/a/11644/2444">this</a> or <a href="https://ai.stackexchange.com/a/24362/2444">this</a> answers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T11:53:21.163Z
Answers.LastActivityDate: 2021-11-29T11:53:21.163Z
Answers.UserDisplayName: nbro
Body: <p>So the code is related to using a buffer</p>
  <pre><code>class BufferWrapper(gym.ObservationWrapper):
      def __init__(self, env, n_steps, dtype=np.int):
          super(BufferWrapper, self).__init__(env)
          self.dtype = dtype
          old_space = env.observation_space
          self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0),
                                                  old_space.high.repeat(n_steps, axis=0), dtype=dtype)
  
      def reset(self):
          self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype)
          return self.observation(self.env.reset())
  
      def observation(self, observation):
          self.buffer[:-1] = self.buffer[1:]
          self.buffer[-1] = observation
          return self.buffer
  </code></pre>
  <p>It is used to basically do some image processing so that the DQN is fed some transformation of the image. This <a href="https://towardsdatascience.com/deep-q-network-dqn-i-bce08bdf2af" rel="nofollow noreferrer">link</a> provides some higher-level logic behind some operations.</p>
  <p>How can I actually understand what's the reason behind the code? Almost all repos have the exact same lines with no explanation (e.g. <a href="https://github.com/hizircanbayram/FeDQN-A-Federated-Learning-Approach-for-Training-Reinforcement-Learning-Agent-of-Atari-Games/blob/main/FederatedRLPong.py" rel="nofollow noreferrer">atari games GitHub repo</a>).</p>
  <p>My specific question is what is the purpose of the line <code>self.buffer[-1] = observation</code>?</p>
  <p>In my case, my observation is a (7*1) array and I have to return that in an appropriate manner from the <code>observation</code> function.</p>
  <p><a href="https://pytorch-lightning-bolts.readthedocs.io/_/downloads/en/0.1.1/pdf/" rel="nofollow noreferrer">The book</a> has some mention of this class but I couldn't understand much from it</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><gym><experience-replay>
Title: What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?
UserDisplayName: user3656142
ThreadText: # What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?
  
  (Asked by: username_0 on 2021-11-17)
  
  So the code is related to using a buffer
  class BufferWrapper(gym.ObservationWrapper):
      def __init__(self, env, n_steps, dtype=np.int):
          super(BufferWrapper, self).__init__(env)
          self.dtype = dtype
          old_space = env.observation_space
          self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0),
                                                  old_space.high.repeat(n_steps, axis=0), dtype=dtype)
  
      def reset(self):
          self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype)
          return self.observation(self.env.reset())
  
      def observation(self, observation):
          self.buffer[:-1] = self.buffer[1:]
          self.buffer[-1] = observation
          return self.buffer
  
  It is used to basically do some image processing so that the DQN is fed some transformation of the image. This link provides some higher-level logic behind some operations.
  How can I actually understand what's the reason behind the code? Almost all repos have the exact same lines with no explanation (e.g. atari games GitHub repo).
  My specific question is what is the purpose of the line self.buffer[-1] = observation?
  In my case, my observation is a (7*1) array and I have to return that in an appropriate manner from the observation function.
  The book has some mention of this class but I couldn't understand much from it
  
  **Tags:** reinforcement-learning, deep-rl, dqn, gym, experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-29. Score: 1 | Accepted answer
  
  I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and experience replay), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN.
  Let's take a look at the observation method.
  def observation(self, observation):
      self.buffer[:-1] = self.buffer[1:]
      self.buffer[-1] = observation
      return self.buffer
  
  
  self.buffer[:-1] = self.buffer[1:] essentially drops/forgets the first (which is also the oldest) observation in the buffer.
  
  self.buffer[-1] = observation adds the new observation (passed as parameter) as the last element of the buffer
  
  
  You can execute the following code to see what that method does.
  buffer = [10, 7, 3]
  print(buffer)
  
  buffer[:-1] = buffer[1:]
  
  observation = 5
  buffer[-1] = observation
  
  print(buffer) # [7, 3, 5]
  
  If you are not familiar with the experience replay technique, you can take a look at this or this answers.
  
  
  ---
  
  