Hi,
I have somehow similar problem. i was wondering if you could help me.
I had the same runtime error and ran my code with CUDA_LAUNCH_BLOCKING=1 and here is the output:
Traceback (most recent call last):
File “MINE_main.py”, line 62, in 
result = util.train((x, y), mine_net,mine_net_optim, checkpoint_template, iter_num = iter_num, save_model=False, verbose = True)
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 77, in train
mi_lb, ma_et = learn_mine(batch, mine_net, mine_net_optim, ma_et)
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 32, in learn_mine
mi_lb, t, et = mutual_information(joint, marginal, mine_net)
File “/nas/home/jiazli/Bias_assessment/model/MINE/util.py”, line 18, in mutual_information
t = mine_net(joint)
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl
result = self.forward(*input, **kwargs)
File “/nas/home/jiazli/Bias_assessment/model/MINE/MINE.py”, line 19, in forward
output = F.elu(self.fc1(input))
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl
result = self.forward(*input, **kwargs)
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 91, in forward
return F.linear(input, self.weight, self.bias)
File “/nas/home/jiazli/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py”, line 1674, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)