Answers.Body: <p>So for anyone struggling to understand the <a href="https://spinningup.openai.com" rel="nofollow noreferrer">OpenAI's Spinning Up</a> educational resource, I'll provide the answer to my question here.</p>
  
  <p>Firstly, it's important to understand that the <a href="https://spinningup.openai.com/en/latest/user/algorithms.html" rel="nofollow noreferrer">algorithms</a> expect a 2-dimensional input <strong>shape</strong>, in rudimentary terms a shape of <code>Box(int)</code>, which isn't the case with the default <a href="https://gym.openai.com/envs/Breakout-v0/" rel="nofollow noreferrer">Breakout-v0</a> game environment, which supplies inputs in the shape <code>Box(210, 160, 3)</code>, which is the game screen's height, width, and RGB colour space.</p>
  
  <p>The <a href="https://gym.openai.com/envs/Breakout-ram-v0/" rel="nofollow noreferrer">Breakout-ram-v0</a> game environment, however, <em>DOES</em> provide an input of the appropriate shape, consequently <code>Box(128,)</code>. So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (<a href="https://spinningup.openai.com/en/latest/algorithms/ppo.html" rel="nofollow noreferrer">PPO</a> in my case), the command should be:</p>
  
  <pre><code>python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest
  </code></pre>
  
  <p>To take this a step further, how would you train an off-policy algorithm (such as <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">DDPQ</a>)? Well, the off-policy algorithms expect a <strong>continuous action space</strong>, which can be seen in the differences in the source code here between <a href="https://gym.openai.com/envs/MountainCar-v0/" rel="nofollow noreferrer">MountainCar-v0</a>, and <a href="https://gym.openai.com/envs/MountainCarContinuous-v0/" rel="nofollow noreferrer">MountainCarContinuous-v0</a>:</p>
  
  <pre><code>self.action_space = spaces.Discrete(3)
  </code></pre>
  
  <p>...versus:</p>
  
  <pre><code>self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
      shape=(1,), dtype=np.float32)
  </code></pre>
  
  <p>So, to sum up, how would you train something like the <a href="https://gym.openai.com/envs/LunarLander-v2/" rel="nofollow noreferrer">Lunar Lander</a> game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:</p>
  
  <pre><code>python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200
  </code></pre>
  
  <p><a href="https://i.sstatic.net/Npihs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Npihs.png" alt="LunarLanderContinuous-v2 OpenAI DDPQ test"></a></p>
  
  <p>Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their <a href="https://github.com/openai/gym/wiki/Table-of-environments" rel="nofollow noreferrer">table of game environments</a> for a quick reference to the observation and action spaces. Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T18:17:23.070Z
Answers.LastActivityDate: 2019-07-26T18:17:23.070Z
Answers.UserDisplayName: Chris Kempen
Body: <p>I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.</p>
  
  <p>I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:</p>
  
  <pre><code>python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest
  </code></pre>
  
  <p>...and then the error:</p>
  
  <pre><code>ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
   (op: 'Multinomial') with input shapes: [?,210,160,4], [].
  </code></pre>
  
  <p>I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.</p>
  
  <p>Thank you so much for any help on the seeminly noobish question!</p>
  
Tags: <reinforcement-learning><open-ai>
Title: OpenAI Spinning Up: Breakout-v0 example
UserDisplayName: Chris Kempen
ThreadText: # OpenAI Spinning Up: Breakout-v0 example
  
  (Asked by: username_0 on 2019-07-23)
  
  I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.
  I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:
  python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest
  
  ...and then the error:
  ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
   (op: 'Multinomial') with input shapes: [?,210,160,4], [].
  
  I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.
  Thank you so much for any help on the seeminly noobish question!
  
  **Tags:** reinforcement-learning, open-ai
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-07-26. Score: 1 | Accepted answer
  
  So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here.
  Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in the shape Box(210, 160, 3), which is the game screen's height, width, and RGB colour space.
  The Breakout-ram-v0 game environment, however, DOES provide an input of the appropriate shape, consequently Box(128,). So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (PPO in my case), the command should be:
  python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest
  
  To take this a step further, how would you train an off-policy algorithm (such as DDPQ)? Well, the off-policy algorithms expect a continuous action space, which can be seen in the differences in the source code here between MountainCar-v0, and MountainCarContinuous-v0:
  self.action_space = spaces.Discrete(3)
  
  ...versus:
  self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
      shape=(1,), dtype=np.float32)
  
  So, to sum up, how would you train something like the Lunar Lander game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:
  python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200
  
  
  Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their table of game environments for a quick reference to the observation and action spaces. Good luck!
  
  
  ---
  
  