title: Getting the error "AttributeError: 'NoneType' object has no attribute 'shape'" when implementing Atari Breakout
body: <p>I have made a code to solve Atari Breakout. I am facing a little problem, but I can't say what it is. </p>
  
  <p>Here is the <a href="https://github.com/JeremieGauthier/AI_Exercices/blob/master/Atari_Breakout/DQN_Breakout_with_ptan.py" rel="nofollow noreferrer">code</a> </p>
  
  <p>It is a problem with the replay memory. </p>
  
  <pre><code>try:
      next_states = torch.tensor(batch[3], dtype=torch.float32) 
  except:
      import ipdb; ipdb.set_trace()
  </code></pre>
  
  <p>The problem is located where are those lines.  <code>set_trace()</code> is use to pop-up an interactive shell. From there, if I run <code>for i in range(batch_size): print(batch[3][i].shape)</code>, I obtained this output</p>
  
  <pre><code>ipdb&gt; for i in range(32): print(batch[3][i].shape)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  (4, 84, 84)
  *** AttributeError: 'NoneType' object has no attribute 'shape'
  </code></pre>
  
  <p>How can I improve that code to avoid such error?</p>