Answers.Body: <p>So for anyone struggling to understand the <a href="https://spinningup.openai.com" rel="nofollow noreferrer">OpenAI's Spinning Up</a> educational resource, I'll provide the answer to my question here.</p>
  
  <p>Firstly, it's important to understand that the <a href="https://spinningup.openai.com/en/latest/user/algorithms.html" rel="nofollow noreferrer">algorithms</a> expect a 2-dimensional input <strong>shape</strong>, in rudimentary terms a shape of <code>Box(int)</code>, which isn't the case with the default <a href="https://gym.openai.com/envs/Breakout-v0/" rel="nofollow noreferrer">Breakout-v0</a> game environment, which supplies inputs in the shape <code>Box(210, 160, 3)</code>, which is the game screen's height, width, and RGB colour space.</p>
  
  <p>The <a href="https://gym.openai.com/envs/Breakout-ram-v0/" rel="nofollow noreferrer">Breakout-ram-v0</a> game environment, however, <em>DOES</em> provide an input of the appropriate shape, consequently <code>Box(128,)</code>. So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (<a href="https://spinningup.openai.com/en/latest/algorithms/ppo.html" rel="nofollow noreferrer">PPO</a> in my case), the command should be:</p>
  
  <pre><code>python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest
  </code></pre>
  
  <p>To take this a step further, how would you train an off-policy algorithm (such as <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">DDPQ</a>)? Well, the off-policy algorithms expect a <strong>continuous action space</strong>, which can be seen in the differences in the source code here between <a href="https://gym.openai.com/envs/MountainCar-v0/" rel="nofollow noreferrer">MountainCar-v0</a>, and <a href="https://gym.openai.com/envs/MountainCarContinuous-v0/" rel="nofollow noreferrer">MountainCarContinuous-v0</a>:</p>
  
  <pre><code>self.action_space = spaces.Discrete(3)
  </code></pre>
  
  <p>...versus:</p>
  
  <pre><code>self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
      shape=(1,), dtype=np.float32)
  </code></pre>
  
  <p>So, to sum up, how would you train something like the <a href="https://gym.openai.com/envs/LunarLander-v2/" rel="nofollow noreferrer">Lunar Lander</a> game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:</p>
  
  <pre><code>python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200
  </code></pre>
  
  <p><a href="https://i.sstatic.net/Npihs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Npihs.png" alt="LunarLanderContinuous-v2 OpenAI DDPQ test"></a></p>
  
  <p>Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their <a href="https://github.com/openai/gym/wiki/Table-of-environments" rel="nofollow noreferrer">table of game environments</a> for a quick reference to the observation and action spaces. Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T18:17:23.070Z
Answers.LastActivityDate: 2019-07-26T18:17:23.070Z
Answers.UserDisplayName: Chris Kempen
Body: <p>I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.</p>
  
  <p>I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:</p>
  
  <pre><code>python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest
  </code></pre>
  
  <p>...and then the error:</p>
  
  <pre><code>ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
   (op: 'Multinomial') with input shapes: [?,210,160,4], [].
  </code></pre>
  
  <p>I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.</p>
  
  <p>Thank you so much for any help on the seeminly noobish question!</p>
  
Tags: <reinforcement-learning><open-ai>
Title: OpenAI Spinning Up: Breakout-v0 example
UserDisplayName: Chris Kempen
ThreadText: # OpenAI Spinning Up: Breakout-v0 example
  
  (Asked by: username_0 on 2019-07-23)
  
  I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.
  
  I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:
  
  ```
  python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest
  
  ```
  
  ...and then the error:
  
  ```
  ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
   (op: 'Multinomial') with input shapes: [?,210,160,4], [].
  
  ```
  
  I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.
  
  Thank you so much for any help on the seeminly noobish question!
  
  **Tags:** reinforcement-learning, open-ai
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-07-26. Score: 1 | Accepted answer
  
  So for anyone struggling to understand the [OpenAI's Spinning Up](https://spinningup.openai.com) educational resource, I'll provide the answer to my question here.
  
  Firstly, it's important to understand that the [algorithms](https://spinningup.openai.com/en/latest/user/algorithms.html) expect a 2-dimensional input **shape**, in rudimentary terms a shape of `Box(int)`, which isn't the case with the default [Breakout-v0](https://gym.openai.com/envs/Breakout-v0/) game environment, which supplies inputs in the shape `Box(210, 160, 3)`, which is the game screen's height, width, and RGB colour space.
  
  The [Breakout-ram-v0](https://gym.openai.com/envs/Breakout-ram-v0/) game environment, however, *DOES* provide an input of the appropriate shape, consequently `Box(128,)`. So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms ([PPO](https://spinningup.openai.com/en/latest/algorithms/ppo.html) in my case), the command should be:
  
  ```
  python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest
  
  ```
  
  To take this a step further, how would you train an off-policy algorithm (such as [DDPQ](https://spinningup.openai.com/en/latest/algorithms/ddpg.html))? Well, the off-policy algorithms expect a **continuous action space**, which can be seen in the differences in the source code here between [MountainCar-v0](https://gym.openai.com/envs/MountainCar-v0/), and [MountainCarContinuous-v0](https://gym.openai.com/envs/MountainCarContinuous-v0/):
  
  ```
  self.action_space = spaces.Discrete(3)
  
  ```
  
  ...versus:
  
  ```
  self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
      shape=(1,), dtype=np.float32)
  
  ```
  
  So, to sum up, how would you train something like the [Lunar Lander](https://gym.openai.com/envs/LunarLander-v2/) game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:
  
  ```
  python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200
  
  ```
  
  [![LunarLanderContinuous-v2 OpenAI DDPQ test](https://i.sstatic.net/Npihs.png)](https://i.sstatic.net/Npihs.png)
  
  Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their [table of game environments](https://github.com/openai/gym/wiki/Table-of-environments) for a quick reference to the observation and action spaces. Good luck!
  
  
  ---
  
  