Answers.Body: <blockquote>
    <p>How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?</p>
  </blockquote>
  
  <p>A Convolutional Neural Network (CNN) structure is a standard neural network architecture when working with 2D pixel input in reinforcement learning, and it is the technique used in the original DQN paper (see paragraphs 1 &amp; 3 of Section 4.1 of <a href="https://arxiv.org/abs/1312.5602" rel="nofollow noreferrer">https://arxiv.org/abs/1312.5602</a>). CNNs typically take 3-dimensional input, where the first two dimensions are <code>height</code> and <code>width</code> of your images, and the third is <code>rgb color</code>. The technique in the paper was to convert each RGB frame (or image) to greyscale format (so it has only 1 color channel/dimension instead of 3) and instead use the <code>rgb_color</code> dimension as a <code>frames</code> dimension that is indexed by each stacked frame.</p>
  
  <blockquote>
    <p>Currently, I am watching a YouTuber: Machine Learning with Phil, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.</p>
  </blockquote>
  
  <p>Later in the tutorial series, he most likely will discuss the training of the neural network. During training, you need to find the q-values of a batch of sets of stacked frames. Specifically, <strong>each element of the batch is a set of stacked frames</strong>. In other words, a set of stacked frames is treated as a single observation, so a batch of sets of stacked frames is a batch of observations.</p>
  
  <p>To find these q-values, you will perform a forward pass of the batch of observations through the neural network. A forward pass of a single observation (set of stacked frames) through the neural network yields a vector of q-values (one for each action). Thus, a forward pass of a batch of observations (batch of stacked frames) will yield a matrix of q-values (one vector of q-values for each observation (or set of stacked frames)). This technique is used because many standard neural network libraries are designed to <strong>perform a forward pass on a batch of inputs through the neural network much faster than performing a forward pass on each input separately</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T20:11:15.250Z
Answers.LastActivityDate: 2020-06-05T21:04:50.037Z
Answers.LastEditDate: 2020-06-05T21:04:50.037Z
Answers.UserDisplayName: DeepQZero
Body: <p>How would you train a reinforcement learning agent from raw pixels? </p>
  
  <p>For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values? </p>
  
  <p>If you pass that batch output, it would be a batch of values, so from here it is impossible to deduce which ones are the true Q-values for that state.</p>
  
  <p>Currently, I am watching a YouTuber: <a href="https://www.youtube.com/watch?v=RfNxXlO6BiA" rel="nofollow noreferrer">Machine Learning with Phil</a>, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.</p>
  
Tags: <reinforcement-learning><ai-design><deep-rl>
Title: How to train a reinforcement learning agent from raw pixels?
UserDisplayName: dato nefaridze
ThreadText: # How to train a reinforcement learning agent from raw pixels?
  
  (Asked by: username_0 on 2020-06-05)
  
  How would you train a reinforcement learning agent from raw pixels?
  
  For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?
  
  If you pass that batch output, it would be a batch of values, so from here it is impossible to deduce which ones are the true Q-values for that state.
  
  Currently, I am watching a YouTuber: [Machine Learning with Phil](https://www.youtube.com/watch?v=RfNxXlO6BiA), and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.
  
  **Tags:** reinforcement-learning, ai-design, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-05. Score: 3 | Accepted answer
  
  > How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?
  
  A Convolutional Neural Network (CNN) structure is a standard neural network architecture when working with 2D pixel input in reinforcement learning, and it is the technique used in the original DQN paper (see paragraphs 1 & 3 of Section 4.1 of <https://arxiv.org/abs/1312.5602>). CNNs typically take 3-dimensional input, where the first two dimensions are `height` and `width` of your images, and the third is `rgb color`. The technique in the paper was to convert each RGB frame (or image) to greyscale format (so it has only 1 color channel/dimension instead of 3) and instead use the `rgb_color` dimension as a `frames` dimension that is indexed by each stacked frame.
  
  > Currently, I am watching a YouTuber: Machine Learning with Phil, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.
  
  Later in the tutorial series, he most likely will discuss the training of the neural network. During training, you need to find the q-values of a batch of sets of stacked frames. Specifically, **each element of the batch is a set of stacked frames**. In other words, a set of stacked frames is treated as a single observation, so a batch of sets of stacked frames is a batch of observations.
  
  To find these q-values, you will perform a forward pass of the batch of observations through the neural network. A forward pass of a single observation (set of stacked frames) through the neural network yields a vector of q-values (one for each action). Thus, a forward pass of a batch of observations (batch of stacked frames) will yield a matrix of q-values (one vector of q-values for each observation (or set of stacked frames)). This technique is used because many standard neural network libraries are designed to **perform a forward pass on a batch of inputs through the neural network much faster than performing a forward pass on each input separately**.
  
  
  ---
  
  