Answers.Body: <p>If you only need 5 frames to predict the next frame, then I'd recommend a <a href="https://arxiv.org/abs/1505.04597" rel="nofollow noreferrer">U-Net architecture</a>, wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.</p>
  
  <p>If using a longer history of frames can help, then I recommend taking a look at <a href="https://arxiv.org/abs/1704.02254" rel="nofollow noreferrer">"Recurrent Environment Simulators"</a> and combining it with the ideas above.</p>
  
  <p>Hope it helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T14:13:54.123Z
Answers.Text: Thanks a lot for your answer @Mehran Shakerinava, I can't bump up your answer yet becuase my reputation is under 15, but still thanks. Can I adapt U-net architectures for tasks other than image segmentation?
Answers.UserDisplayName: lr99
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T14:35:47.927Z
Answers.Text: @lr99 You're welcome! The U-Net architecture can be used to produce any image. In image segmentation, this output image is suggesting how the image can be segmented, but in general it could be anything.
Answers.UserDisplayName: Mehran Shakerinava
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T12:59:57.770Z
Answers.LastActivityDate: 2019-09-01T12:59:57.770Z
Answers.UserDisplayName: Mehran Shakerinava
Body: <p>I am working on an image to image regression task which requires me to develop a deep learning model that takes in a sequence of 5 images and return another image. The sequence of 5 images and the output images are conceptually and temporally related. In fact, the 5 images in the sequence each correspond to timestep in a simulation and the output, the one I am trying to predict, corresponds eventually to the 6th timestep of that sequence.</p>
  
  <p>For now, I have been training a simple regression-type CNN model which takes in the sequence of 5 images stored in a list and outputs an image corresponding to the next timestep in the simulation. This does work with a small and rather simple dataset (13000 images) but works a bit worse on a more diverse and larger dataset (102000 images). </p>
  
  <p>For this reason, I have been researching a bit now in order to find a better way to carry out this task and I found the idea of ConvLSTMs. However, I have seen these applied to the prediction of feature and the output of a sentence describing that image. What I wanted to know is whether ConvLSTMs can also output images, but more importantly if they can be applied to my case. If not, what other types of deep learning network can be suitable for this task?</p>
  
  <p>Thanks in advance!</p>
  
Tags: <deep-learning><python>
Title: Can ConvLSTMs outuput images?
UserDisplayName: lr99
ThreadText: # Can ConvLSTMs outuput images?
  
  (Asked by: username_0 on 2019-08-30)
  
  I am working on an image to image regression task which requires me to develop a deep learning model that takes in a sequence of 5 images and return another image. The sequence of 5 images and the output images are conceptually and temporally related. In fact, the 5 images in the sequence each correspond to timestep in a simulation and the output, the one I am trying to predict, corresponds eventually to the 6th timestep of that sequence.
  
  For now, I have been training a simple regression-type CNN model which takes in the sequence of 5 images stored in a list and outputs an image corresponding to the next timestep in the simulation. This does work with a small and rather simple dataset (13000 images) but works a bit worse on a more diverse and larger dataset (102000 images).
  
  For this reason, I have been researching a bit now in order to find a better way to carry out this task and I found the idea of ConvLSTMs. However, I have seen these applied to the prediction of feature and the output of a sentence describing that image. What I wanted to know is whether ConvLSTMs can also output images, but more importantly if they can be applied to my case. If not, what other types of deep learning network can be suitable for this task?
  
  Thanks in advance!
  
  **Tags:** deep-learning, python
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-01. Score: 2
  
  If you only need 5 frames to predict the next frame, then I'd recommend a [U-Net architecture](https://arxiv.org/abs/1505.04597), wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.
  
  If using a longer history of frames can help, then I recommend taking a look at ["Recurrent Environment Simulators"](https://arxiv.org/abs/1704.02254) and combining it with the ideas above.
  
  Hope it helps!
  
  > **Comments:**
  > **username_0**: Thanks a lot for your answer @username_1, I can't bump up your answer yet becuase my reputation is under 15, but still thanks. Can I adapt U-net architectures for tasks other than image segmentation?  
  > **username_1**: @username_0 You're welcome! The U-Net architecture can be used to produce any image. In image segmentation, this output image is suggesting how the image can be segmented, but in general it could be anything.  
  
  ---
  
  