<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Sorry if a duplicate of existing bug-report (can't easily find it)</p>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.zeros((16*2**20 - 512)//2 + 1, 1, dtype=torch.float16, device='cuda:0') @ torch.zeros(1, 2, dtype=torch.float16, device='cuda:0')"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>((<span class="pl-c1">16</span><span class="pl-c1">*</span><span class="pl-c1">2</span><span class="pl-c1">**</span><span class="pl-c1">20</span> <span class="pl-c1">-</span> <span class="pl-c1">512</span>)<span class="pl-c1">//</span><span class="pl-c1">2</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span>) @ <span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>(<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span>)</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCBlas.cu:315"><pre class="notranslate"><code class="notranslate">RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCBlas.cu:315
</code></pre></div>
<p dir="auto">It also fails if cublasGemmEx called via pytorch extension, with stacktrace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7fe8d5a44dc5 in /opt/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: mm_fp16_fp16_acc32(at::Tensor, at::Tensor, at::Tensor) + 0x7dd (0x7fe884d2873d in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #2: &lt;unknown function&gt; + 0x199f0 (0x7fe884d389f0 in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #3: &lt;unknown function&gt; + 0x19eda (0x7fe884d38eda in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #4: &lt;unknown function&gt; + 0x16a79 (0x7fe884d35a79 in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #5: _PyMethodDef_RawFastCallKeywords + 0x264 (0x5632b1b5f494 in /opt/anaconda3/bin/python)
frame #6: _PyCFunction_FastCallKeywords + 0x21 (0x5632b1b5f5b1 in /opt/anaconda3/bin/python)
frame #7: _PyEval_EvalFrameDefault + 0x4f32 (0x5632b1bbb5b2 in /opt/anaconda3/bin/python)
frame #8: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #9: PyEval_EvalCodeEx + 0x44 (0x5632b1afc8a4 in /opt/anaconda3/bin/python)
frame #10: PyEval_EvalCode + 0x1c (0x5632b1afc8cc in /opt/anaconda3/bin/python)
frame #11: &lt;unknown function&gt; + 0x1e140d (0x5632b1bc640d in /opt/anaconda3/bin/python)
frame #12: _PyMethodDef_RawFastCallKeywords + 0xe9 (0x5632b1b5f319 in /opt/anaconda3/bin/python)
frame #13: _PyCFunction_FastCallKeywords + 0x21 (0x5632b1b5f5b1 in /opt/anaconda3/bin/python)
frame #14: _PyEval_EvalFrameDefault + 0x47e3 (0x5632b1bbae63 in /opt/anaconda3/bin/python)
frame #15: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #16: _PyFunction_FastCallKeywords + 0x325 (0x5632b1b5e765 in /opt/anaconda3/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #18: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #19: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #20: _PyEval_EvalFrameDefault + 0x14e3 (0x5632b1bb7b63 in /opt/anaconda3/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0xb99 (0x5632b1afc2d9 in /opt/anaconda3/bin/python)
frame #22: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #25: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #26: _PyEval_EvalFrameDefault + 0x14e3 (0x5632b1bb7b63 in /opt/anaconda3/bin/python)
frame #27: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #29: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #30: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #31: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #32: _PyFunction_FastCallKeywords + 0x325 (0x5632b1b5e765 in /opt/anaconda3/bin/python)
frame #33: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #34: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #35: _PyEval_EvalFrameDefault + 0x466 (0x5632b1bb6ae6 in /opt/anaconda3/bin/python)
frame #36: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #37: _PyEval_EvalFrameDefault + 0x4b54 (0x5632b1bbb1d4 in /opt/anaconda3/bin/python)
frame #38: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #39: PyEval_EvalCodeEx + 0x44 (0x5632b1afc8a4 in /opt/anaconda3/bin/python)
frame #40: PyEval_EvalCode + 0x1c (0x5632b1afc8cc in /opt/anaconda3/bin/python)
frame #41: &lt;unknown function&gt; + 0x230f24 (0x5632b1c15f24 in /opt/anaconda3/bin/python)
frame #42: PyRun_FileExFlags + 0xa1 (0x5632b1c20111 in /opt/anaconda3/bin/python)
frame #43: PyRun_SimpleFileExFlags + 0x1c3 (0x5632b1c20303 in /opt/anaconda3/bin/python)
frame #44: &lt;unknown function&gt; + 0x23c41b (0x5632b1c2141b in /opt/anaconda3/bin/python)
frame #45: _Py_UnixMain + 0x3c (0x5632b1c214fc in /opt/anaconda3/bin/python)
frame #46: __libc_start_main + 0xf5 (0x7fe8f24b33d5 in /lib64/libc.so.6)
frame #47: &lt;unknown function&gt; + 0x1e14e2 (0x5632b1bc64e2 in /opt/anaconda3/bin/python)"><pre class="notranslate"><code class="notranslate">frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7fe8d5a44dc5 in /opt/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: mm_fp16_fp16_acc32(at::Tensor, at::Tensor, at::Tensor) + 0x7dd (0x7fe884d2873d in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #2: &lt;unknown function&gt; + 0x199f0 (0x7fe884d389f0 in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #3: &lt;unknown function&gt; + 0x19eda (0x7fe884d38eda in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #4: &lt;unknown function&gt; + 0x16a79 (0x7fe884d35a79 in /mnt/live/research/hffx/SharedCodes/python/xtx/nn/gemm_11/gemm_bind.cpython-37m-x86_64-linux-gnu.so)
frame #5: _PyMethodDef_RawFastCallKeywords + 0x264 (0x5632b1b5f494 in /opt/anaconda3/bin/python)
frame #6: _PyCFunction_FastCallKeywords + 0x21 (0x5632b1b5f5b1 in /opt/anaconda3/bin/python)
frame #7: _PyEval_EvalFrameDefault + 0x4f32 (0x5632b1bbb5b2 in /opt/anaconda3/bin/python)
frame #8: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #9: PyEval_EvalCodeEx + 0x44 (0x5632b1afc8a4 in /opt/anaconda3/bin/python)
frame #10: PyEval_EvalCode + 0x1c (0x5632b1afc8cc in /opt/anaconda3/bin/python)
frame #11: &lt;unknown function&gt; + 0x1e140d (0x5632b1bc640d in /opt/anaconda3/bin/python)
frame #12: _PyMethodDef_RawFastCallKeywords + 0xe9 (0x5632b1b5f319 in /opt/anaconda3/bin/python)
frame #13: _PyCFunction_FastCallKeywords + 0x21 (0x5632b1b5f5b1 in /opt/anaconda3/bin/python)
frame #14: _PyEval_EvalFrameDefault + 0x47e3 (0x5632b1bbae63 in /opt/anaconda3/bin/python)
frame #15: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #16: _PyFunction_FastCallKeywords + 0x325 (0x5632b1b5e765 in /opt/anaconda3/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #18: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #19: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #20: _PyEval_EvalFrameDefault + 0x14e3 (0x5632b1bb7b63 in /opt/anaconda3/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0xb99 (0x5632b1afc2d9 in /opt/anaconda3/bin/python)
frame #22: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #25: _PyFunction_FastCallKeywords + 0x387 (0x5632b1b5e7c7 in /opt/anaconda3/bin/python)
frame #26: _PyEval_EvalFrameDefault + 0x14e3 (0x5632b1bb7b63 in /opt/anaconda3/bin/python)
frame #27: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #29: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #30: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #31: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #32: _PyFunction_FastCallKeywords + 0x325 (0x5632b1b5e765 in /opt/anaconda3/bin/python)
frame #33: _PyEval_EvalFrameDefault + 0x6f5 (0x5632b1bb6d75 in /opt/anaconda3/bin/python)
frame #34: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #35: _PyEval_EvalFrameDefault + 0x466 (0x5632b1bb6ae6 in /opt/anaconda3/bin/python)
frame #36: _PyFunction_FastCallKeywords + 0xfb (0x5632b1b5e53b in /opt/anaconda3/bin/python)
frame #37: _PyEval_EvalFrameDefault + 0x4b54 (0x5632b1bbb1d4 in /opt/anaconda3/bin/python)
frame #38: _PyEval_EvalCodeWithName + 0x2f8 (0x5632b1afba38 in /opt/anaconda3/bin/python)
frame #39: PyEval_EvalCodeEx + 0x44 (0x5632b1afc8a4 in /opt/anaconda3/bin/python)
frame #40: PyEval_EvalCode + 0x1c (0x5632b1afc8cc in /opt/anaconda3/bin/python)
frame #41: &lt;unknown function&gt; + 0x230f24 (0x5632b1c15f24 in /opt/anaconda3/bin/python)
frame #42: PyRun_FileExFlags + 0xa1 (0x5632b1c20111 in /opt/anaconda3/bin/python)
frame #43: PyRun_SimpleFileExFlags + 0x1c3 (0x5632b1c20303 in /opt/anaconda3/bin/python)
frame #44: &lt;unknown function&gt; + 0x23c41b (0x5632b1c2141b in /opt/anaconda3/bin/python)
frame #45: _Py_UnixMain + 0x3c (0x5632b1c214fc in /opt/anaconda3/bin/python)
frame #46: __libc_start_main + 0xf5 (0x7fe8f24b33d5 in /lib64/libc.so.6)
frame #47: &lt;unknown function&gt; + 0x1e14e2 (0x5632b1bc64e2 in /opt/anaconda3/bin/python)
</code></pre></div>
<p dir="auto">It all works as expected for fp32, it also works for fp16 <em>if</em> matrix is slightly smaller:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.zeros((16*2**20 - 512)//2, 1, dtype=torch.float16, device='cuda:0') @ torch.zeros(1, 2, dtype=torch.float16, device='cuda:0')"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>((<span class="pl-c1">16</span><span class="pl-c1">*</span><span class="pl-c1">2</span><span class="pl-c1">**</span><span class="pl-c1">20</span> <span class="pl-c1">-</span> <span class="pl-c1">512</span>)<span class="pl-c1">//</span><span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span>) @ <span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>(<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span>)</pre></div>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch: 1.1.0</li>
<li>OS: Linux</li>
<li>Python version: 3.7</li>
<li>CUDA/cuDNN version: 10.0.130</li>
<li>GPU: V-100</li>
</ul>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Error using the pytorch.hub API:<br>
code:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="entrypoints = torch.hub.list('pytorch/vision', force_reload=True)"><pre class="notranslate"><code class="notranslate">entrypoints = torch.hub.list('pytorch/vision', force_reload=True)
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Downloading: &quot;https://github.com/pytorch/vision/archive/master.zip&quot; to .../.cache\torch\hub\master.zip
Traceback (most recent call last):
  File &quot;..../test.py&quot;, line 2, in &lt;module&gt;
    entrypoints = torch.hub.list('pytorch/vision', force_reload=True)
  File &quot;...\.conda\envs\hackaton\lib\site-packages\torch\hub.py&quot;, line 285, in list
    hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + MODULE_HUBCONF)
  File &quot;..\.conda\envs\hackaton\lib\site-packages\torch\hub.py&quot;, line 72, in import_module
    spec.loader.exec_module(module)
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 678, in exec_module
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 219, in _call_with_frames_removed
  File &quot;.../.cache\torch\hub\pytorch_vision_master/hubconf.py&quot;, line 4, in &lt;module&gt;
    from torchvision.models.alexnet import alexnet
  File &quot;..../.cache\torch\hub\pytorch_vision_master\torchvision\__init__.py&quot;, line 1, in &lt;module&gt;
    from torchvision import models
  File &quot;...../.cache\torch\hub\pytorch_vision_master\torchvision\models\__init__.py&quot;, line 6, in &lt;module&gt;
    from .densenet import *
  File &quot;...../.cache\torch\hub\pytorch_vision_master\torchvision\models\densenet.py&quot;, line 22, in &lt;module&gt;
    class _DenseLayer(nn.Module):
  File &quot;.../.cache\torch\hub\pytorch_vision_master\torchvision\models\densenet.py&quot;, line 52, in _DenseLayer
    @torch.jit.unused  # noqa: T484
AttributeError: module 'torch.jit' has no attribute 'unused'

Process finished with exit code 1"><pre class="notranslate"><code class="notranslate">Downloading: "https://github.com/pytorch/vision/archive/master.zip" to .../.cache\torch\hub\master.zip
Traceback (most recent call last):
  File "..../test.py", line 2, in &lt;module&gt;
    entrypoints = torch.hub.list('pytorch/vision', force_reload=True)
  File "...\.conda\envs\hackaton\lib\site-packages\torch\hub.py", line 285, in list
    hub_module = import_module(MODULE_HUBCONF, repo_dir + '/' + MODULE_HUBCONF)
  File "..\.conda\envs\hackaton\lib\site-packages\torch\hub.py", line 72, in import_module
    spec.loader.exec_module(module)
  File "&lt;frozen importlib._bootstrap_external&gt;", line 678, in exec_module
  File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed
  File ".../.cache\torch\hub\pytorch_vision_master/hubconf.py", line 4, in &lt;module&gt;
    from torchvision.models.alexnet import alexnet
  File "..../.cache\torch\hub\pytorch_vision_master\torchvision\__init__.py", line 1, in &lt;module&gt;
    from torchvision import models
  File "...../.cache\torch\hub\pytorch_vision_master\torchvision\models\__init__.py", line 6, in &lt;module&gt;
    from .densenet import *
  File "...../.cache\torch\hub\pytorch_vision_master\torchvision\models\densenet.py", line 22, in &lt;module&gt;
    class _DenseLayer(nn.Module):
  File ".../.cache\torch\hub\pytorch_vision_master\torchvision\models\densenet.py", line 52, in _DenseLayer
    @torch.jit.unused  # noqa: T484
AttributeError: module 'torch.jit' has no attribute 'unused'

Process finished with exit code 1
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ailzhang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ailzhang">@ailzhang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/suo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/suo">@suo</a></p>