Ok, nice. Could you explain a bit more about your dataset, i.e.

the image sequence size
where does your data come from (natural images, medical images?)
batch_size

Are you normalizing the input? If not, you should try it.
If you are normalizing, try a tanh in your model output.
Your output might be just “out of bounds”, which could make the training quite hard.