The following is the error. However, I do not get the error when I run my programming with the four Tesla V100s. How can I solve the problem? Thank you!
Start epoch: 0
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File “main.py”, line 166, in 
p_img.copy_(netG(p_z).detach())
File “/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 477, in call
result = self.forward(*input, **kwargs)
File “/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py”, line 100, in forward
out,p1 = self.attn1(out)
File “/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 477, in call
result = self.forward(*input, **kwargs)
File “/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py”, line 32, in forward
energy =  torch.bmm(proj_query,proj_key) # transpose check
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCBlas.cu:411