I see this has been posted a few times before, but none of those responses have helped.
From here 17, I’m trying:
CUDA_LAUNCH_BLOCKING=1 && export CUDA_VISIBLE_DEVICES=0 && python train_transformer_encoder.py --batch_size 1

But I’m getting:
Traceback (most recent call last):
  File "train_transformer_encoder.py", line 207, in <module>
    main()
  File "train_transformer_encoder.py", line 129, in main
    train_loss = train(reconstruct_spect_model, optimizer=optimizer, dataset=train_loader)
  File "train_transformer_encoder.py", line 194, in train
    pred = model(x)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/speech-reconstruction/models/transformer_reconstruct.py", line 45, in forward
    src = self.inp_embedding(src)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

So I’ve tried a batch_size of 1 and the CUDA_LAUNCH_BLOCKING as well, but no dice. Any help would be greatly appreciated.
Thank you in advance