title: pytorch with GPU, CUBLAS_STATUS_EXECUTION_FAILED error, how to trace or fix it?
body: <p>I'm a newbie to GPU training and pytorch.</p>
  <p>While executing a NMT code, got CUBLAS_STATUS_EXECUTION_FAILED error. (of course, with CPU, It performed well..)  MY GPU is not so excellent, I know, but the error is occuring every time with 1 batch size.</p>
  <p>Please help me to fix or trace this.</p>
  <pre><code>Epoch [1/30]:   1%|▍                                                                    Epoch [1/30]:   1%|                 | 1/166 [00:00&lt;?, ?it/s, loss=7.72, ppl=2.24e+3, |g_Epoch [1/30]:   1%|                 | 1/166 [00:00&lt;?, ?it/s, loss=7.72, ppl=2.24e+3, |g_Epoch [1/30]:   1%|                 | 1/166 [00:00&lt;?, ?it/s, loss=7.72, ppl=2.25e+3, |g_Epoch [1/30]:   1%|         | 2/166 [00:00&lt;00:22,  7.43it/s, loss=7.72, ppl=2.25e+3, |g_Epoch [1/30]:   1%|         | 2/166 [00:00&lt;00:22,  7.39it/s, loss=7.72, ppl=2.25e+3, |g_Epoch [1/30]:   2%|▏        | 3/166 [00:00&lt;00:11, 14.74it/s, loss=7.72, ppl=2.25e+3, |g_Epoch [1/30]:   2%|▏        | 3/166 [00:00&lt;00:11, 14.74it/s, loss=7.72, ppl=2.25e+3, |g_Epoch [1/30]:   2%|▏        | 3/166 [00:00&lt;00:11, 14.74it/s, loss=7.72, ppl=2.26e+3, |g_Epoch [1/30]:   2%|▏        | 4/166 [00:00&lt;00:10, 14.74it/s, loss=7.72, ppl=2.26e+3, |g_Epoch [1/30]:   2%|▏        | 4/166 [00:00&lt;00:10, 14.74it/s, loss=7.72, ppl=2.26e+3, |g_Epoch [1/30]:   3%|▎        | 5/166 [00:00&lt;00:13, 12.01it/s, loss=7.72, ppl=2.26e+3, |g_param|=4.27e+5, |param|=1.29e+3]
  </code></pre>
  <pre><code>Current run is terminating due to exception: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &amp;falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &amp;fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
  </code></pre>
  <pre><code> File &quot;/home/00/pythonp/python38/pytorch/lib/python3.8/site-packages/torch/nn/modules/linear.py&quot;, line 94, in forward
      return F.linear(input, self.weight, self.bias)
    File &quot;/home/00/pythonp/python38/pytorch/lib/python3.8/site-packages/torch/nn/functional.py&quot;, line 1753, in linear
      return torch._C._nn.linear(input, weight, bias)
  RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &amp;falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &amp;fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
  </code></pre>
  <p>My GPU is..</p>
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  Quadro P620         On   | 00000000:81:00.0 Off |                  N/A |
  | 34%   42C    P8    N/A /  N/A |     19MiB /  1999MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    0   N/A  N/A      2662      G   /usr/lib/xorg/Xorg                  9MiB |
  |    0   N/A  N/A      3026      G   /usr/bin/gnome-shell                4MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p>With 'CUBLAS_STATUS_EXECUTION_FAILED' keyword, there're many advices to upgrade  CUDA Version to 11.2. However, It's already 11.2.</p>
  <p>GPU Memory Usage..</p>
  <pre><code>| 34%   41C    P8    N/A /  N/A |     19MiB /  1999MiB |      0%      Default |
  | 34%   41C    P0    N/A /  N/A |     64MiB /  1999MiB |      2%      Default |
  | 34%   42C    P0    N/A /  N/A |    202MiB /  1999MiB |      4%      Default |
  | 34%   43C    P0    N/A /  N/A |    346MiB /  1999MiB |      3%      Default |
  | 34%   43C    P0    N/A /  N/A |    498MiB /  1999MiB |      3%      Default |
  | 34%   43C    P0    N/A /  N/A |    632MiB /  1999MiB |      3%      Default |
  | 34%   43C    P0    N/A /  N/A |    850MiB /  1999MiB |      2%      Default |
  | 34%   44C    P0    N/A /  N/A |    962MiB /  1999MiB |     47%      Default |
  | 34%   44C    P0    N/A /  N/A |    962MiB /  1999MiB |     45%      Default |
  | 34%   45C    P0    N/A /  N/A |     19MiB /  1999MiB |     19%      Default |
  </code></pre>
  <p>...</p>