@SimonW, @richard  upgraded to pytorch 0.4.0a0+02b758f and it continues to run out of memory on a 16GB gpu and batch size 32, whereas with pytorch 0.3 it could run the same 32 batch size on a smaller 12 gb GPU.
It is able to go past the decoder steps but fails in subsequent operations, more specifically when computing dropout.