This is a good question, and I think the memory issues you’re seeing are the result of the way PyTorch currently wraps cuDNN LSTMs (which is listed as an issue but probably won’t be fixed for a little while). Try it with torch.backends.cudnn.enabled = False and see if you can fit a larger hidden size.