I have installed the pytorch 1.5.0 with CUDA 10.2 and cudnn 7.6.5.32.
It still gives a run time error after about 2000 iterations. But this time it gives another error message as follow. This time it seems happens when a torch.cat operation is implementing.
RuntimeError: cuda runtime error (719) : unspecified launch failure at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278
Traceback (most recent call last):
File “train.py”, line 426, in 
hparams,
File “train.py”, line 284, in train
y_pred_t_list = model(x, teacher=True)
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/server/workspace/projects/tacotron2/model.py”, line 664, in forward
teacher=teacher,
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/server/workspace/projects/tacotron2/model.py”, line 520, in forward
) = self.decode(decoder_input)
File “/home/server/workspace/projects/tacotron2/model.py”, line 465, in decode
decoder_input = torch.cat((prenet_output, self.attention_context), -1)