Answers.Body: <p>I reviewed some videos of Amidar on YouTube, and it seems that the game screen is fixed for a few seconds before the start of each level. This is probably to give the human player a chance to rest between levels and maybe even pause the game to take a break, as the gameplay looks very intense.</p>
  <blockquote>
  <p>How do I get this environment to move immediately?</p>
  </blockquote>
  <p>I suggest creating a gym <a href="https://github.com/openai/gym/tree/master/gym/wrappers" rel="nofollow noreferrer">wrapper</a> to change the <code>reset</code> function of the environment to produce a different start state. Since the initial 85 frames are not influenced by the agent's actions, these frames are unhelpful and unnecessary for your agent's training. You could consider calling the <code>step</code> function 85 times in your new implementation of the <code>reset</code> function using a random action; then return the resultant state. When calling your newly implemented <code>reset</code> function, the agent's start state will be frame 85 instead of frame 0. As you noted, frame 85 is immediately followed with many frames of meaningful movement, which will provide higher-quality training data. In contrast, the original implementation of the <code>reset</code> function starts at frame 0, which is followed by 85 unhelpful fixed frames.</p>
  <blockquote>
  <p>Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything.</p>
  </blockquote>
  <p>ATARI environments are difficult even for some of the more powerful RL algorithms. For example, it is one of the harder environments for DQN to solve (see the Nature <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">paper</a>, pg. 531).  Don't worry if your agent isn't learning right away. It could take hours or even days to learn enough to noticeably improve, even with a correct implementation of a powerful algorithm. Good luck, and welcome to AI Stack Exchange!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T18:31:38.467Z
Answers.LastActivityDate: 2021-05-24T18:31:38.467Z
Answers.UserDisplayName: DeepQZero
Body: <p>When I try to run <a href="https://gym.openai.com/envs/Amidar-v0/" rel="nofollow noreferrer">Amidar</a> even without RL code, I cannot get the environment to move immediately. It takes about 100 steps before the game actually starts moving. I use the following simple code to display some images and print some actions (I always try to do the same action, namely going up):</p>
  <pre><code>env = gym.make('Amidar-v0')
  env.reset()
  
  for i in range(1000):
      action = 2 
      next_state, reward, terminated, info = env.step(action) # take a random action
      print(f&quot;Timestep {i}&quot;)
      print(next_state.shape)
      print(reward)
      print(action)
      print(info)
      plt.imshow(next_state)
      plt.show()
  </code></pre>
  <p>When running this code, it takes until about step 85 before the environment starts to move. After that, each step, it moves until the agent is hit by the enemy. Then the environment restarts in the start state, and it takes quite some time before it starts to move again. I have tried doing 'FIRE' as my first action; however, this is not working since it also takes a while before the environment starts moving. Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything. How do I get this environment to move immediately?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-25T18:57:26.503Z
Comments.Text: Hi @Lennart and welcome to AI Stack Exchange! It looks like this question was closed because it is off-topic. If you would like to keep this question open, you will need to edit it to meet the on-topic rules discussed [here](https://ai.stackexchange.com/help/on-topic). Just comment here if you would like help in any way with this. I hope to see you around the site!
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T19:24:33.900Z
Comments.Text: Hi @DeepQZero, thanks for the response. I just read the off-topic rules and it was probably closed because the question is about specific software. No need to keep this question open.
Comments.UserDisplayName: Lennart
Tags: <reinforcement-learning><open-ai><gym><atari-games>
Title: Why does the Atari Gym Amidar environment only move after a certain number of episodes?
UserDisplayName: Lennart
ThreadText: # Why does the Atari Gym Amidar environment only move after a certain number of episodes?
  
  (Asked by: username_0 on 2021-05-18)
  
  When I try to run Amidar even without RL code, I cannot get the environment to move immediately. It takes about 100 steps before the game actually starts moving. I use the following simple code to display some images and print some actions (I always try to do the same action, namely going up):
  env = gym.make('Amidar-v0')
  env.reset()
  
  for i in range(1000):
      action = 2 
      next_state, reward, terminated, info = env.step(action) # take a random action
      print(f"Timestep {i}")
      print(next_state.shape)
      print(reward)
      print(action)
      print(info)
      plt.imshow(next_state)
      plt.show()
  
  When running this code, it takes until about step 85 before the environment starts to move. After that, each step, it moves until the agent is hit by the enemy. Then the environment restarts in the start state, and it takes quite some time before it starts to move again. I have tried doing 'FIRE' as my first action; however, this is not working since it also takes a while before the environment starts moving. Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything. How do I get this environment to move immediately?
  
  **Tags:** reinforcement-learning, open-ai, gym, atari-games
  > **Comments:**
  > **username_1**: Hi @Lennart and welcome to AI Stack Exchange! It looks like this question was closed because it is off-topic. If you would like to keep this question open, you will need to edit it to meet the on-topic rules discussed [here](https://ai.stackexchange.com/help/on-topic). Just comment here if you would like help in any way with this. I hope to see you around the site!  
  > **username_0**: Hi @DeepQZero, thanks for the response. I just read the off-topic rules and it was probably closed because the question is about specific software. No need to keep this question open.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-24. Score: 0 | Accepted answer
  
  I reviewed some videos of Amidar on YouTube, and it seems that the game screen is fixed for a few seconds before the start of each level. This is probably to give the human player a chance to rest between levels and maybe even pause the game to take a break, as the gameplay looks very intense.
  
  > How do I get this environment to move immediately?
  
  I suggest creating a gym [wrapper](https://github.com/openai/gym/tree/master/gym/wrappers) to change the `reset` function of the environment to produce a different start state. Since the initial 85 frames are not influenced by the agent's actions, these frames are unhelpful and unnecessary for your agent's training. You could consider calling the `step` function 85 times in your new implementation of the `reset` function using a random action; then return the resultant state. When calling your newly implemented `reset` function, the agent's start state will be frame 85 instead of frame 0. As you noted, frame 85 is immediately followed with many frames of meaningful movement, which will provide higher-quality training data. In contrast, the original implementation of the `reset` function starts at frame 0, which is followed by 85 unhelpful fixed frames.
  
  > Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything.
  
  ATARI environments are difficult even for some of the more powerful RL algorithms. For example, it is one of the harder environments for DQN to solve (see the Nature [paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf), pg. 531). Don't worry if your agent isn't learning right away. It could take hours or even days to learn enough to noticeably improve, even with a correct implementation of a powerful algorithm. Good luck, and welcome to AI Stack Exchange!
  
  
  ---
  
  