Answers.Body: <p>What's exactly the point of <code>time.sleep()</code> in this code? I don't really understand it, you're simply stopping the execution of the program for <span class="math-container">$0.01$</span> seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call <code>env.step</code> function and returns the next state and reward. Calling sleep function only slows down the program here.  </p>
  
  <p>The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-27T12:54:15.973Z
Answers.Text: To introduce a deterministic element between two successive runs of the `env.step(action)` to alleviate the noise in the environment if possible.
Answers.UserDisplayName: vikram71198
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-27T13:21:04.827Z
Answers.Text: sleep function will not affect the randomness of the simulator, it's determined by it's own random number generator. See if you can set the seed of the simulator for repeated deterministic episodes.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-27T11:48:02.457Z
Answers.LastActivityDate: 2020-01-27T11:48:02.457Z
Answers.UserDisplayName: Brale
Answers.Body: <p>Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them. </p>
  
  <p>So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the current observation. </p>
  
  <p>Your current setup lends itself well to a case-based reasoning (CBR) approach. The idea of CBR is that you have a memory bank of observation-action pairs and when you see a new observation you look up the memory bank and see if the current observation has been seen before. If so, do that action. The interesting part is when there are no observations that match directly, but there are some that are similar. In this case you choose the most similar. The similarity can be calculated in any number of ways, and it is dependant on the data types. This paper is will provide a good start: <a href="https://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf" rel="nofollow noreferrer">https://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T08:37:50.030Z
Answers.Text: Do you know of any examples of CBR applied to selecting actions based on video frames from a game? Without knowing more, I would be concerned that it will not scale well to that environment. Potentially the OP could reduce the dimensionality of the observations to help apply CBR
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T14:46:55.280Z
Answers.Text: You have a point in that the similarity metric used to compare two images would have to be well thought out. But I imagine it can still be done. The other option like you said would be to potentially apply convolutions to the images to extract relevant features. Or do other image processing as a feature extraction technique.
Answers.UserDisplayName: Sacha Gunaratne
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-27T19:15:36.687Z
Answers.LastActivityDate: 2020-01-27T19:15:36.687Z
Answers.UserDisplayName: Sacha Gunaratne
Body: <p>I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is defined as the case when the agent is atleast able to successfully retrieve the key (Gets a reward of 100). And such cases I dump in a pickle file. I got 44 successful cases when I kept it to run for a day or so. Here is the code I use to generate the training set :</p>
  
  <pre><code>import numpy
  import gym
  import cma
  import random
  import time
  import pickle as pkl
  
  env   = gym.make('MontezumaRevenge-ram-v0')
  
  observation = env.reset()
  
  #print(observation)
  #print(env.action_space.sample())
  
  obs_dict    = []
  action_dict = []
  success_ctr = 0
  
  for i in range(0, 1000000):
      print('Reward for episode',i+1)
      done = False
      rew = 0
      action_list = []
      obs_list    = []
      while not done:
          action = env.action_space.sample()
          observation, reward, done, _ = env.step(action)
          action_list.append(action)
          obs_list.append(observation)
  
          rew += reward
          env.render()
          time.sleep(0.01)
          if done:
              env.reset()
              if rew &gt; 0:
                  success_ctr += 1
                  print(action_list)
                  action_dict.append(action_list)
                  obs_dict.append(obs_list)
                  pkl.dump(obs_dict, open("obslist.pkl", "wb"))
                  pkl.dump(action_dict, open("action.pkl", "wb"))
  
      print(rew)
      time.sleep(1)
  
  try:
      print(obs_dict.shape)
  except:
      pass
  
  print("Took key:", success_ctr)
  </code></pre>
  
  <p>I loaded the successful cases from my generated pickle file, and simulated the agent's playing using those exact same cases. But, the agent never receives a reward of 100. I dont understand why this is happening. A little search online suggested it could be because of noise in the game. So, I gave a sleep time, before running each episode. Still, doesn't work. Can someone please explain why is this happening? And suggest a way I could go about generating the training set?</p>
  
Tags: <reinforcement-learning><open-ai><rewards><gym>
Title: Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful
UserDisplayName: vikram71198
ThreadText: # Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful
  
  (Asked by: username_0 on 2020-01-27)
  
  I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is defined as the case when the agent is atleast able to successfully retrieve the key (Gets a reward of 100). And such cases I dump in a pickle file. I got 44 successful cases when I kept it to run for a day or so. Here is the code I use to generate the training set :
  import numpy
  import gym
  import cma
  import random
  import time
  import pickle as pkl
  
  env   = gym.make('MontezumaRevenge-ram-v0')
  
  observation = env.reset()
  
  #print(observation)
  #print(env.action_space.sample())
  
  obs_dict    = []
  action_dict = []
  success_ctr = 0
  
  for i in range(0, 1000000):
      print('Reward for episode',i+1)
      done = False
      rew = 0
      action_list = []
      obs_list    = []
      while not done:
          action = env.action_space.sample()
          observation, reward, done, _ = env.step(action)
          action_list.append(action)
          obs_list.append(observation)
  
          rew += reward
          env.render()
          time.sleep(0.01)
          if done:
              env.reset()
              if rew > 0:
                  success_ctr += 1
                  print(action_list)
                  action_dict.append(action_list)
                  obs_dict.append(obs_list)
                  pkl.dump(obs_dict, open("obslist.pkl", "wb"))
                  pkl.dump(action_dict, open("action.pkl", "wb"))
  
      print(rew)
      time.sleep(1)
  
  try:
      print(obs_dict.shape)
  except:
      pass
  
  print("Took key:", success_ctr)
  
  I loaded the successful cases from my generated pickle file, and simulated the agent's playing using those exact same cases. But, the agent never receives a reward of 100. I dont understand why this is happening. A little search online suggested it could be because of noise in the game. So, I gave a sleep time, before running each episode. Still, doesn't work. Can someone please explain why is this happening? And suggest a way I could go about generating the training set?
  
  **Tags:** reinforcement-learning, open-ai, rewards, gym
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-01-27. Score: 2
  
  What's exactly the point of `time.sleep()` in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call `env.step` function and returns the next state and reward. Calling sleep function only slows down the program here.
  
  The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.
  
  > **Comments:**
  > **username_0**: To introduce a deterministic element between two successive runs of the `env.step(action)` to alleviate the noise in the environment if possible.  
  > **username_1**: sleep function will not affect the randomness of the simulator, it's determined by it's own random number generator. See if you can set the seed of the simulator for repeated deterministic episodes.  
  
  ---
  
  ### Answer by username_2 on 2020-01-27. Score: 2
  
  Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them.
  
  So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the current observation.
  
  Your current setup lends itself well to a case-based reasoning (CBR) approach. The idea of CBR is that you have a memory bank of observation-action pairs and when you see a new observation you look up the memory bank and see if the current observation has been seen before. If so, do that action. The interesting part is when there are no observations that match directly, but there are some that are similar. In this case you choose the most similar. The similarity can be calculated in any number of ways, and it is dependant on the data types. This paper is will provide a good start: <https://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf>
  
  > **Comments:**
  > **username_3**: Do you know of any examples of CBR applied to selecting actions based on video frames from a game? Without knowing more, I would be concerned that it will not scale well to that environment. Potentially the OP could reduce the dimensionality of the observations to help apply CBR  
  > **username_2**: You have a point in that the similarity metric used to compare two images would have to be well thought out. But I imagine it can still be done. The other option like you said would be to potentially apply convolutions to the images to extract relevant features. Or do other image processing as a feature extraction technique.  
  
  ---
  
  