You could either call .train() on the nn.RNN, as cudnn needs this layer in training mode to calculate the gradients (and disable dropout if necessary in this layer via model.rnn.dropout = 0.0) or disable cudnn for this layer via:
x = ...
with torch.backends.cudnn.flags(enabled=False):
    x = self.rnn(x)
x = ...