Your first error still looks like not properly caught out of memory error. What likely happens is workspace tensor cannot be properly allocated at https://github.com/pytorch/pytorch/blob/master/torch/backends/cudnn/rnn.py#L280-L281 9 and then when workspace.size(0) is called, out of range error is reported.
Usually if you can get through 1 iteration, memory usage should not grow, so I’m not sure what happens. Are you properly detaching all your variables so that backpropagation graph does not grow indefinitely?