I think it would affect the gradients (assuming you could disable cudnn for the RNN), since e.g. dropout layers won’t be used and the running stats in batchnorm layers are applied to normalize the activation input instead of the batch statistics.