ptrblck:

Also, could you post the complete error message from the last post?


i suppose the complete error message is as follows？and when the code is running，it takes about 9000M while the whole GPU memory is about 11000M。So, it might not run out of memory. I’ll still have a try to use half of the current batch_size and check GPU memory again.
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=719 : unspecified launch failure
Traceback (most recent call last):
File “train.py”, line 426, in 
hparams,
File “train.py”, line 284, in train
y_pred_t_list = model(x, teacher=True)
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/server/workspace/projects/tacotron2/model.py”, line 666, in forward
teacher=teacher,
File “/home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/server/workspace/projects/tacotron2/model.py”, line 522, in forward
) = self.decode(decoder_input)
File “/home/server/workspace/projects/tacotron2/model.py”, line 449, in decode
dim=1,
RuntimeError: cuda runtime error (719) : unspecified launch failure at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278