I met a quite confused problem when running the https://github.com/NVIDIA/tacotron2.git 14
with ：
a single rtx 2080ti GPU
pytorch verision 1.2.0
cuda version 10.0
cudnn version 7.6.4
The problem happens suddenly after hundreds of iterations’ training，usually 800~2000 steps.
Sometimes it gives a error message just like in the title. Some times it gives nothing and just seems stuck. When it gives the runtime error, i found the problem happened when the code implements a matrix multiplication in a nn.Linear layer (input multiply weight). The input and weight tensors are all FLOAT32 when i debugging.
I have searched this problem on internet and find that it is similar to a bug of FP16 operation which is solved with CUDA 10.1. But, when i update the cuda to 10.1 and pytorch to 1.5.0, it still happened. I also checked my code, all the tensors are FLOAT32 and no FP16 operation.
The same code has also been running on a 1080ti GPU with same cuda, cudnn, and pytorch version. And everything is OK, it seems that it happened only when i use a 2080ti GPU. Could anyone give some advices to avoid this problem?