I don’t believe that LSTM is acting “weird”, but that cuDNN might have issues with its internal handler creation. Do you see the same issue if you run the script on the second GPU (while GPU:0 is fully allocated) via CUDA_VISIBLE_DEVICES=1 python script.py and use cuda:0 inside the script?