Answers.Body: <p>Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-11T15:33:48.933Z
Answers.LastActivityDate: 2018-01-11T15:33:48.933Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>I've read through a few papers on next frame prediction from a sequence of frames and several of them use spatial transformations (<a href="https://arxiv.org/abs/1506.02025" rel="nofollow noreferrer">STNs</a>). See <a href="https://arxiv.org/abs/1605.08104" rel="nofollow noreferrer">this</a> as an example. I want to know what are the pros and cons of using an STN to predict the next frame. Are there any assumptions that must be made about the data besides "Consecutive frames are all approximately affine transformations of each other"?</p>
  
Tags: <neural-networks><deep-learning><image-recognition><deep-neural-networks><prediction>
Title: What are the pros and cons of using a spatial transformation network to predict the next video frame?
UserDisplayName: Jaden Travnik
ThreadText: # What are the pros and cons of using a spatial transformation network to predict the next video frame?
  
  (Asked by: username_0 on 2017-06-07)
  
  I've read through a few papers on next frame prediction from a sequence of frames and several of them use spatial transformations ([STNs](https://arxiv.org/abs/1506.02025)). See [this](https://arxiv.org/abs/1605.08104) as an example. I want to know what are the pros and cons of using an STN to predict the next frame. Are there any assumptions that must be made about the data besides "Consecutive frames are all approximately affine transformations of each other"?
  
  **Tags:** neural-networks, deep-learning, image-recognition, deep-neural-networks, prediction
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-01-11. Score: 1 | Accepted answer
  
  Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence.
  
  
  ---
  
  