Answers.Body: <p>In the article <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">Playing Atari with Deep Reinforcement Learning</a>, <em>Mnih et al, 2013</em>, which was a major outbreak in Deep Reinforcement learning (especially in Deep Q learning), they don't feed only the last image to the network. They stack the 4 last images :</p>
  <blockquote>
  <p>For the experiments in this paper, the function φ from algorithm 1
  applies this preprocessing to the last 4 frames of a history and stacks
  them to produce the input to the Q-function</p>
  </blockquote>
  <p>So they add the motion through sequentiality. From various articles and own coding experiences, this seems to me to be the main common approach. I don't know if other techniques have been implemented.</p>
  <p>One thing we could imagine would be to compute the Cross-correlation between a previous frame and the last one, and then feed the cross correlation product to the net.</p>
  <p>Another idea would be to train previously a CNN to extract motion features from a sequence of frames, and feed these extracted features to your net. <a href="https://www.duo.uio.no/bitstream/handle/10852/58552/2/ArticlePIVANN.pdf" rel="nofollow noreferrer">This article (Performing Particle Image Velocimetry using Artificial Neural Networks: a proof-of-concept)</a>, <em>Rabault et al, 2017</em> is an example of a CNN to extract motion features.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:44:02.280Z
Answers.Text: Another alternative is to use a RNN and feed it the frames one at a time. It's not what the original Atari DQN did, but has been tried successfully: http://cs229.stanford.edu/proj2016/report/ChenYingLaird-DeepQLearningWithRecurrentNeuralNetwords-report.pdf
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:47:46.170Z
Answers.Text: Obviously, how could I not mention it ? thank you !
Answers.UserDisplayName: 16Aghnar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T12:50:52.573Z
Answers.LastActivityDate: 2020-08-25T12:57:11.813Z
Answers.LastEditDate: 2020-08-25T12:57:11.813Z
Answers.UserDisplayName: 16Aghnar
Body: <p>Basic deep reinforcement learning methods use as input an image for the current state, do some convolutions on that image, apply some reinforcement learning algorithm, and it is solved.</p>
  <p>Let us take the game Breakout or Pong as an example. What I do not understand is: how does the agent understand when an object is moving towards it or away from it? I believe that the action it chooses must be different in these two scenarios, and, from a single image as input, there is no notion of motion.</p>
  
Tags: <reinforcement-learning><deep-rl><data-preprocessing><atari-games>
Title: How does the RL agent understand motion if it gets only one image as input?
UserDisplayName: devidduma
ThreadText: # How does the RL agent understand motion if it gets only one image as input?
  
  (Asked by: username_0 on 2020-08-25)
  
  Basic deep reinforcement learning methods use as input an image for the current state, do some convolutions on that image, apply some reinforcement learning algorithm, and it is solved.
  
  Let us take the game Breakout or Pong as an example. What I do not understand is: how does the agent understand when an object is moving towards it or away from it? I believe that the action it chooses must be different in these two scenarios, and, from a single image as input, there is no notion of motion.
  
  **Tags:** reinforcement-learning, deep-rl, data-preprocessing, atari-games
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-25. Score: 3 | Accepted answer
  
  In the article [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/pdf/1312.5602.pdf), *Mnih et al, 2013*, which was a major outbreak in Deep Reinforcement learning (especially in Deep Q learning), they don't feed only the last image to the network. They stack the 4 last images :
  
  > For the experiments in this paper, the function φ from algorithm 1
  > applies this preprocessing to the last 4 frames of a history and stacks
  > them to produce the input to the Q-function
  
  So they add the motion through sequentiality. From various articles and own coding experiences, this seems to me to be the main common approach. I don't know if other techniques have been implemented.
  
  One thing we could imagine would be to compute the Cross-correlation between a previous frame and the last one, and then feed the cross correlation product to the net.
  
  Another idea would be to train previously a CNN to extract motion features from a sequence of frames, and feed these extracted features to your net. [This article (Performing Particle Image Velocimetry using Artificial Neural Networks: a proof-of-concept)](https://www.duo.uio.no/bitstream/handle/10852/58552/2/ArticlePIVANN.pdf), *Rabault et al, 2017* is an example of a CNN to extract motion features.
  
  > **Comments:**
  > **username_2**: Another alternative is to use a RNN and feed it the frames one at a time. It's not what the original Atari DQN did, but has been tried successfully: http://cs229.stanford.edu/proj2016/report/ChenYingLaird-DeepQLearningWithRecurrentNeuralNetwords-report.pdf  
  > **username_1**: Obviously, how could I not mention it ? thank you !  
  
  ---
  
  