This is a limitation of the CuDNN implementation (combined with the layering of autograd over ATen, which does not allow the CuDNN implementation in ATen to see whether gradients are required). An easy way is to fix this is to set the entire network to eval but then put the RNNs into train (this re-enables dropout you might have in them, though). Another way is to disable the CuDNN backend (with torch.backends.cudnn.flags(enabled=False): ...) or use JITed RNNs 3.
Best regards
Thomas