Answers.Body: <p>I read the DQN paper titled: Playing Atari with Deep Reinforcement Learning again</p>
  
  <p>I read, in the pre-processing and model architecture section (section 4.1), that for each state that is input to the CNN, that this state is actually stacked frames of the game, so basically what has to be done, to my understanding, is that for each time step you stack 4 frames (current frame and 3 previous frames) and this will serve as input to the CNN as the dimensions would be side * side * 4, 4 because the frames are converted to grey-scale and 4 frames are being used. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T13:00:29.423Z
Answers.Text: A related question has been asked recently and the given answer, [https://ai.stackexchange.com/a/21673/2444](https://ai.stackexchange.com/a/21673/2444), seems to be consistent with your understanding.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T10:23:02.933Z
Answers.LastActivityDate: 2020-06-07T13:09:00.493Z
Answers.LastEditDate: 2020-06-07T13:09:00.493Z
Answers.UserDisplayName: rert588
Answers.Body: <blockquote>
    <p>I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN? </p>
  </blockquote>
  
  <p>As nbro mentioned in a comment to your answer, this question has very recently been asked and answered <a href="https://ai.stackexchange.com/a/21673/37607">here</a>.</p>
  
  <blockquote>
    <p>I also want to know since <span class="math-container">$s_1$</span> is equal to a set, which can be seen in line 4 of the algorithm, then why is <span class="math-container">$s_{t+1}$</span> equal to <span class="math-container">$s_t$</span>, <span class="math-container">$a_t$</span>, <span class="math-container">$x_{t+1}$</span>?</p>
  </blockquote>
  
  <p>The algorithm presented in the original DQN <a href="https://arxiv.org/abs/1312.5602" rel="nofollow noreferrer">paper</a> is relatively simple and written to express the main ideas of their approach (e.g. experience replay, preprocessing histories, gradient descent, etc.); in fact, it isn't even the exact algorithm used in the experiments! For example, the experiments use frame-skipping to reduce computation - this is not mentioned in Algorithm 1 in the paper. With that in mind, setting <span class="math-container">$s_{t+1}$</span> equal to <span class="math-container">$s_t, a_t, x_{t+1}$</span> in the algorithm signifies a <strong>general notion</strong> of constructing the next raw state <span class="math-container">$s_{t+1}$</span> from the previous preprocessed state <span class="math-container">$s_t$</span>, previous action <span class="math-container">$a_t$</span>, and current frame <span class="math-container">$x_{t+1}$</span>. For example:</p>
  
  <ul>
  <li>If the action space at the next timestep is constrained by the state, then the state may need additional parameters to encode the action space.</li>
  <li>The algorithm needs some indication if a state is terminal, and such an indication may need to be encoded in the state.</li>
  <li>If there is frame skipping, then multiple frames will be needed to construct the next state, possibly using the previous state as well.</li>
  </ul>
  
  <p>The above examples should display how <strong>the encoding of the state cannot always simply be a stack of raw frames, or even a function of <span class="math-container">$s_t$</span>, <span class="math-container">$a_t$</span> and <span class="math-container">$x_{t+1}$</span>, and therefore a more general approach is often required</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T21:28:09.570Z
Answers.LastActivityDate: 2020-06-07T21:28:09.570Z
Answers.UserDisplayName: DeepQZero
Body: <p>I recently read the DQN <a href="https://arxiv.org/abs/1312.5602" rel="nofollow noreferrer">paper</a> titled: Playing Atari with Deep Reinforcement Learning. My basic and rough understanding of the paper is as follows:</p>
  
  <p>You have two neural networks; one stays frozen for a duration of time steps and is used in the computation of the loss function with the neural network that is updating. The loss function is used to update the neural network using gradient descent.</p>
  
  <p>Experience replay is used, which basically creates a buffer of experiences. This buffer of experiences is randomly sampled and these random samples are used to update the non-frozen neural network.</p>
  
  <p>My question pertains to the DQN algorithm illustrated in the paper: Algorithm 1, more specifically lines 4 and 9 of this algorithm. My understanding, which is also mentioned early on in the paper, is that the states are actually sequences of the game-play frames. I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN? </p>
  
  <p>I also want to know since <span class="math-container">$s_{1}$</span> is equal to a set, which can be seen in line 4 of the algorithm, then why is <span class="math-container">$s_{t+1}$</span> equal to <span class="math-container">$s_{t}$</span>, <span class="math-container">$a_{t}$</span>, <span class="math-container">$x_{t+1}$</span>?</p>
  
Tags: <neural-networks><reinforcement-learning><convolutional-neural-networks><dqn>
Title: How to convert sequences of images into state in DQN?
UserDisplayName: rert588
ThreadText: # How to convert sequences of images into state in DQN?
  
  (Asked by: username_0 on 2020-06-06)
  
  I recently read the DQN [paper](https://arxiv.org/abs/1312.5602) titled: Playing Atari with Deep Reinforcement Learning. My basic and rough understanding of the paper is as follows:
  
  You have two neural networks; one stays frozen for a duration of time steps and is used in the computation of the loss function with the neural network that is updating. The loss function is used to update the neural network using gradient descent.
  
  Experience replay is used, which basically creates a buffer of experiences. This buffer of experiences is randomly sampled and these random samples are used to update the non-frozen neural network.
  
  My question pertains to the DQN algorithm illustrated in the paper: Algorithm 1, more specifically lines 4 and 9 of this algorithm. My understanding, which is also mentioned early on in the paper, is that the states are actually sequences of the game-play frames. I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN?
  
  I also want to know since $s\_{1}$ is equal to a set, which can be seen in line 4 of the algorithm, then why is $s\_{t+1}$ equal to $s\_{t}$, $a\_{t}$, $x\_{t+1}$?
  
  **Tags:** neural-networks, reinforcement-learning, convolutional-neural-networks, dqn
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-06-07. Score: 3 | Accepted answer
  
  > I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN?
  
  As username_1 mentioned in a comment to your answer, this question has very recently been asked and answered [here](https://ai.stackexchange.com/a/21673/37607).
  
  > I also want to know since $s\_1$ is equal to a set, which can be seen in line 4 of the algorithm, then why is $s\_{t+1}$ equal to $s\_t$, $a\_t$, $x\_{t+1}$?
  
  The algorithm presented in the original DQN [paper](https://arxiv.org/abs/1312.5602) is relatively simple and written to express the main ideas of their approach (e.g. experience replay, preprocessing histories, gradient descent, etc.); in fact, it isn't even the exact algorithm used in the experiments! For example, the experiments use frame-skipping to reduce computation - this is not mentioned in Algorithm 1 in the paper. With that in mind, setting $s\_{t+1}$ equal to $s\_t, a\_t, x\_{t+1}$ in the algorithm signifies a **general notion** of constructing the next raw state $s\_{t+1}$ from the previous preprocessed state $s\_t$, previous action $a\_t$, and current frame $x\_{t+1}$. For example:
  
  * If the action space at the next timestep is constrained by the state, then the state may need additional parameters to encode the action space.
  * The algorithm needs some indication if a state is terminal, and such an indication may need to be encoded in the state.
  * If there is frame skipping, then multiple frames will be needed to construct the next state, possibly using the previous state as well.
  
  The above examples should display how **the encoding of the state cannot always simply be a stack of raw frames, or even a function of $s\_t$, $a\_t$ and $x\_{t+1}$, and therefore a more general approach is often required**.
  
  
  ---
  
  ### Answer by username_0 on 2020-06-07. Score: 1
  
  I read the DQN paper titled: Playing Atari with Deep Reinforcement Learning again
  
  I read, in the pre-processing and model architecture section (section 4.1), that for each state that is input to the CNN, that this state is actually stacked frames of the game, so basically what has to be done, to my understanding, is that for each time step you stack 4 frames (current frame and 3 previous frames) and this will serve as input to the CNN as the dimensions would be side \* side \* 4, 4 because the frames are converted to grey-scale and 4 frames are being used.
  
  > **Comments:**
  > **username_1**: A related question has been asked recently and the given answer, [https://ai.stackexchange.com/a/21673/2444](https://ai.stackexchange.com/a/21673/2444), seems to be consistent with your understanding.  
  
  ---
  
  