Answers.Body: <p>I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and <strong>experience replay</strong>), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN.</p>
  <p>Let's take a look at the <code>observation</code> method.</p>
  <pre><code>def observation(self, observation):
      self.buffer[:-1] = self.buffer[1:]
      self.buffer[-1] = observation
      return self.buffer
  </code></pre>
  <ul>
  <li><p><code>self.buffer[:-1] = self.buffer[1:]</code> essentially drops/forgets the first (which is also the oldest) observation in the buffer.</p>
  </li>
  <li><p><code>self.buffer[-1] = observation</code> adds the new <code>observation</code> (passed as parameter) as the last element of the buffer</p>
  </li>
  </ul>
  <p>You can execute the following code to see what that method does.</p>
  <pre><code>buffer = [10, 7, 3]
  print(buffer)
  
  buffer[:-1] = buffer[1:]
  
  observation = 5
  buffer[-1] = observation
  
  print(buffer) # [7, 3, 5]
  </code></pre>
  <p>If you are not familiar with the <strong>experience replay</strong> technique, you can take a look at <a href="https://ai.stackexchange.com/a/11644/2444">this</a> or <a href="https://ai.stackexchange.com/a/24362/2444">this</a> answers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T11:53:21.163Z
Answers.LastActivityDate: 2021-11-29T11:53:21.163Z
Answers.UserDisplayName: nbro
Body: <p>So the code is related to using a buffer</p>
  <pre><code>class BufferWrapper(gym.ObservationWrapper):
      def __init__(self, env, n_steps, dtype=np.int):
          super(BufferWrapper, self).__init__(env)
          self.dtype = dtype
          old_space = env.observation_space
          self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0),
                                                  old_space.high.repeat(n_steps, axis=0), dtype=dtype)
  
      def reset(self):
          self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype)
          return self.observation(self.env.reset())
  
      def observation(self, observation):
          self.buffer[:-1] = self.buffer[1:]
          self.buffer[-1] = observation
          return self.buffer
  </code></pre>
  <p>It is used to basically do some image processing so that the DQN is fed some transformation of the image. This <a href="https://towardsdatascience.com/deep-q-network-dqn-i-bce08bdf2af" rel="nofollow noreferrer">link</a> provides some higher-level logic behind some operations.</p>
  <p>How can I actually understand what's the reason behind the code? Almost all repos have the exact same lines with no explanation (e.g. <a href="https://github.com/hizircanbayram/FeDQN-A-Federated-Learning-Approach-for-Training-Reinforcement-Learning-Agent-of-Atari-Games/blob/main/FederatedRLPong.py" rel="nofollow noreferrer">atari games GitHub repo</a>).</p>
  <p>My specific question is what is the purpose of the line <code>self.buffer[-1] = observation</code>?</p>
  <p>In my case, my observation is a (7*1) array and I have to return that in an appropriate manner from the <code>observation</code> function.</p>
  <p><a href="https://pytorch-lightning-bolts.readthedocs.io/_/downloads/en/0.1.1/pdf/" rel="nofollow noreferrer">The book</a> has some mention of this class but I couldn't understand much from it</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><gym><experience-replay>
Title: What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?
UserDisplayName: user3656142
ThreadText: # What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?
  
  (Asked by: username_0 on 2021-11-17)
  
  So the code is related to using a buffer
  
  ```
  class BufferWrapper(gym.ObservationWrapper):
      def __init__(self, env, n_steps, dtype=np.int):
          super(BufferWrapper, self).__init__(env)
          self.dtype = dtype
          old_space = env.observation_space
          self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0),
                                                  old_space.high.repeat(n_steps, axis=0), dtype=dtype)
  
      def reset(self):
          self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype)
          return self.observation(self.env.reset())
  
      def observation(self, observation):
          self.buffer[:-1] = self.buffer[1:]
          self.buffer[-1] = observation
          return self.buffer
  
  ```
  
  It is used to basically do some image processing so that the DQN is fed some transformation of the image. This [link](https://towardsdatascience.com/deep-q-network-dqn-i-bce08bdf2af) provides some higher-level logic behind some operations.
  
  How can I actually understand what's the reason behind the code? Almost all repos have the exact same lines with no explanation (e.g. [atari games GitHub repo](https://github.com/hizircanbayram/FeDQN-A-Federated-Learning-Approach-for-Training-Reinforcement-Learning-Agent-of-Atari-Games/blob/main/FederatedRLPong.py)).
  
  My specific question is what is the purpose of the line `self.buffer[-1] = observation`?
  
  In my case, my observation is a (7\*1) array and I have to return that in an appropriate manner from the `observation` function.
  
  [The book](https://pytorch-lightning-bolts.readthedocs.io/_/downloads/en/0.1.1/pdf/) has some mention of this class but I couldn't understand much from it
  
  **Tags:** reinforcement-learning, deep-rl, dqn, gym, experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-29. Score: 1 | Accepted answer
  
  I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and **experience replay**), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN.
  
  Let's take a look at the `observation` method.
  
  ```
  def observation(self, observation):
      self.buffer[:-1] = self.buffer[1:]
      self.buffer[-1] = observation
      return self.buffer
  
  ```
  
  * `self.buffer[:-1] = self.buffer[1:]` essentially drops/forgets the first (which is also the oldest) observation in the buffer.
  * `self.buffer[-1] = observation` adds the new `observation` (passed as parameter) as the last element of the buffer
  
  You can execute the following code to see what that method does.
  
  ```
  buffer = [10, 7, 3]
  print(buffer)
  
  buffer[:-1] = buffer[1:]
  
  observation = 5
  buffer[-1] = observation
  
  print(buffer) # [7, 3, 5]
  
  ```
  
  If you are not familiar with the **experience replay** technique, you can take a look at [this](https://ai.stackexchange.com/a/11644/2444) or [this](https://ai.stackexchange.com/a/24362/2444) answers.
  
  
  ---
  
  