Okay, finally I’ve tried reproduce this issue. The script is shown as following:
import torch

test_lstm_tensor = torch.rand(64,256,768).to('cuda:1')

lstm_layer = torch.nn.LSTM(input_size=768,
                                     hidden_size=768,
                                     num_layers=1,
                                       dropout=0.5,
                                         bidirectional=False,
                                       batch_first=True)
lstm_layer.to('cuda:1')

lstm_layer(test_lstm_tensor)

All tests are performing on cuda:1. And the gpu memory cuda:0 device has already filled by another program. Here’s the nvtop screen:

微信截图_202201080217521920×863 20.9 KB

On these basis, first I run python test_lstm.py command. And the trace log is:
Traceback (most recent call last):
  File "test_lstm.py", line 13, in <module>
    lstm_layer(test_lstm_tensor)
  File "/home/lawbda/envs/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/lawbda/envs/anaconda3/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 679, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

And the I run CUDA_VISIBLE_DEVICES=1 python test_lstm.py, and the trace log is show like:
Traceback (most recent call last):
  File "test_lstm.py", line 3, in <module>
    test_lstm_tensor = torch.rand(64,256,768).to('cuda:1')
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

As stated above, I suspect that the LSTM will call the wrong device when it runs. When I stop the program running on cuda:0 and re-run the test.
Command CUDA_VISIBLE_DEVICES=1 python test_lstm.py will cause error:
Traceback (most recent call last):
  File "test_lstm.py", line 3, in <module>
    test_lstm_tensor = torch.rand(64,256,768).to('cuda:1')
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

And when I run the same code on jupyter notebook, the nvtop will show this:

微信截图_20220108023158958×854 22 KB

This looks like a device selection or jupyter notebook issue, but I really don’t know what the cause of the error is.