Now I know when the problem will occur, and I have some guesses of the problem.
Let me formulate my problem. Normally, I like to plot the output of the deep learning model and the label to see whether the model’s behavior is normal.
Yet I plot the picture before computing the loss.
Here’s the code which will meet the problem.
epochs = 600
clipping_value = 1

model = Net(
    input_dim=1,
    hidden_dim=128,
    layer_dim=5,
    output_dim=1,
    bidirectional=True).cuda()

# torch.cuda.set_device(0)
torch.backends.cudnn.benchmark = True
def get_learning_rate(optimizer):
    lr=[]
    for param_group in optimizer.param_groups:
          lr +=[ param_group['lr'] ]
    return lr

def train_model(epochs, model, learning_rate=0.001, interval_print=100):
    
    if torch.cuda.is_available():
        model = model.cuda()    
    torch.backends.cudnn.enabled = True
    model.train()
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.8)
    criterion = nn.MSELoss()
    x = torch.randn(128, 800, 1).cuda()
    y = torch.randn(128, 800, 1).cuda()
    
    for epoch in range(epochs):
        print("the current epoch is: ", epoch)
        scheduler.step()               
        rnn_out = model(x)
        plt.plot(y.cpu().detach().numpy()[-1, :], 'b-', label= 'label')
        plt.plot(rnn_out.cpu().detach().numpy()[-1, :], 'go', label='output')
        plt.legend(loc='best')
        plt.show()
        plt.plot(x.cpu().detach().numpy()[-1, :, 0], 'g-', label='input')
        plt.show() 
        
        
        loss = criterion( rnn_out ,y)     # try to prevent the loss suface from being too coarse
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), clipping_value)
        optimizer.step() 
            
train_model(epochs, model)        

After a few minutes, cuDNN error will be met.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-e410bf903d5b> in <module>
     52 
     53 
---> 54 train_model(epochs, model)

<ipython-input-4-e410bf903d5b> in train_model(epochs, model, learning_rate, interval_print)
     44         loss = criterion( rnn_out ,y)     # try to prevent the loss suface from being too coarse
     45         optimizer.zero_grad()
---> 46         loss.backward()
     47         torch.nn.utils.clip_grad_norm_(model.parameters(), clipping_value)
     48         optimizer.step()

C:\ProgramData\Anaconda3\envs\pytorch15\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--> 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199 
    200     def register_hook(self, hook):

C:\ProgramData\Anaconda3\envs\pytorch15\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     96         retain_graph = create_graph
     97 
---> 98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
    100         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR (_cudnn_rnn_backward_input at ..\aten\src\ATen\native\cudnn\RNN.cpp:931)
(no backtrace available)

If I skip the plot part of code or plot the picture after computing loss and loss.backward(), the code can run normally.
I suspect that the problem occurs because input, model’s output and label go to cpu during plotting, and when computing the loss loss = criterion( rnn_out ,y) and loss.backward(), error somehow appear.
I only know when the problem will appear yet I still don’t know why it appears. Moreover, such an error doesn’t exist before I update few packages in Anaconda(unfortunately I didn’t preserve detail of the previous environment )
So I wonder if you can give some advice about why such an error occurs. Any advice will help.
Thanks.