Answers.Body: <p>Training happens once you have a result. If the result is good (maybe you won in pong, or you improved your highscore in breakout) all the actions in the game are "supported" by backpropagation, if the result is bad, all the actions in the game are suppressed. </p>
  
  <p>This sounds weird because in each game regardless of the end result you'll have many good and bad actions, but it works if you keep it up for many thousands of games. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-10T17:42:11.033Z
Answers.LastActivityDate: 2016-09-10T17:42:11.033Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p>I know that deepmind used deep Q learning (<a href="https://deepmind.com/research/dqn/" rel="nofollow">DQN</a>) for its Atari game AI. It used a <a href="https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf" rel="nofollow">conv neural network</a> (CNN) to approximate <code>Q(s,a)</code> from pixels instead of from a Q-table. I want to know how DQN converted input to an action. How many output did the CNN have? How did they train the neural network for prediction?</p>
  
  <p>Here are the steps that I believe are happening inside DQN:</p>
  
  <blockquote>
    <p>1) A game picture (a state) is send to CNN as input value</p>
    
    <p>2) CNN predicts an output as action (eg:left, right, shoot, etc)</p>
    
    <p>3) Simulator applies the predicted action and moves to new game state</p>
    
    <p>4) repeat step 1</p>
  </blockquote>
  
  <p>The problem with my above logic is in <strong>step 2</strong>. CNN is used for predicting an action, but when is CNN trained for prediction? </p>
  
  <p>I would prefer if you used less math for explanation.</p>
  
  <p>EDIT</p>
  
  <p>I want to add some more questions regarding the same topic</p>
  
  <p>1) How reward is passed in the neural network? that is how neural network knows whether its output action obtained positive or negative reward?</p>
  
  <p>2) How many output the neural network has and how action is determined from those outputs?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-09-11T10:44:37.157Z
Comments.Text: We prefer only one question per post. If you have more questions, please [ask](http://ai.stackexchange.com/questions/ask) them in a different post.
Comments.UserDisplayName: Mithical
Tags: <deep-neural-networks><deep-learning><convolutional-neural-networks><gaming>
Title: How does deepmind's Atari game AI work?
UserDisplayName: Eka
ThreadText: # How does deepmind's Atari game AI work?
  
  (Asked by: username_0 on 2016-09-10)
  
  I know that deepmind used deep Q learning ([DQN](https://deepmind.com/research/dqn/)) for its Atari game AI. It used a [conv neural network](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) (CNN) to approximate `Q(s,a)` from pixels instead of from a Q-table. I want to know how DQN converted input to an action. How many output did the CNN have? How did they train the neural network for prediction?
  
  Here are the steps that I believe are happening inside DQN:
  
  > 1) A game picture (a state) is send to CNN as input value
  >
  > 2) CNN predicts an output as action (eg:left, right, shoot, etc)
  >
  > 3) Simulator applies the predicted action and moves to new game state
  >
  > 4) repeat step 1
  
  The problem with my above logic is in **step 2**. CNN is used for predicting an action, but when is CNN trained for prediction?
  
  I would prefer if you used less math for explanation.
  
  EDIT
  
  I want to add some more questions regarding the same topic
  
  1) How reward is passed in the neural network? that is how neural network knows whether its output action obtained positive or negative reward?
  
  2) How many output the neural network has and how action is determined from those outputs?
  
  **Tags:** deep-neural-networks, deep-learning, convolutional-neural-networks, gaming
  
  > **Comments:**
  > **username_1**: We prefer only one question per post. If you have more questions, please [ask](http://ai.stackexchange.com/questions/ask) them in a different post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2016-09-10. Score: 3
  
  Training happens once you have a result. If the result is good (maybe you won in pong, or you improved your highscore in breakout) all the actions in the game are "supported" by backpropagation, if the result is bad, all the actions in the game are suppressed.
  
  This sounds weird because in each game regardless of the end result you'll have many good and bad actions, but it works if you keep it up for many thousands of games.
  
  
  ---
  
  