Thanks for sharing your code and for taking the time to help, Thomas!  The code in your notebook is a bit different from what’s in the example as it is right now.  Apart from being CUDA-enabled (which is great!), in yours, both the “outputs” lines in the forward part of the Sequence class look like this:
outputs += [c_t2]
but in the PyTorch/examples entry 5, the corresponding lines read
outputs += [h_t2]
…Not sure how significant that is; I’m still learning.
When I run your code, the intermediary images look like things are working, and the loss gets really small (like 3e-5)…

Screen Shot 2017-11-08 at 4.51.37 PM.jpg1492×640 266 KB

but shortly after that, the Loss begins increasing until the final value is around 0.56 and the final image looks like…

Screen Shot 2017-11-08 at 4.46.40 PM.png1454×572 46.3 KB

Just for definiteness: I’m running Python 3.5 via the anaconda distribution, with CUDA 8.0 and CUDNN 5.1.10 on Ubuntu 16.04.   What are you using?
Thanks again.