I’m not sure which error is raised on which device, as you’ve mentioned device 0 in both error messages.
Based on the first error, your are running into an indexing issue. Rerun the code with CUDA_LAUNCH_BLOCKING=1 python script.pt args and check which operation fails. Once you’ve isolated it, check the indices and make sure they are valid.
After this is fixed, rerun the code on the other devices and check, if the cuDNN error is still raised.
CUDNN_STATUS_INTERNAL_ERROR is also raised, if cuDNN is running into a sticky error caused by a previously failed operation. Since you were seeing the invalid indexing assert, cuDNN might just reraise it.