<p dir="auto">There appears to be a race condition in the index CUDA kernels where multiple threads are writing to the same address.</p>
<p dir="auto">Reproducer code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch

ref = torch.arange(0, 20).view(4, 5).float()

indexer= [slice(None), [1 for i in range(5)]]

src = torch.randperm(ref[indexer].numel()).view(ref[indexer].size()).float()

print('ref', ref)
print('indexer', indexer)
print('ref[indxer]', ref[indexer])
print('src', src)

cpu_ans = ref.clone()
cpu_ans[indexer]=src

nump_ans = ref.double().numpy()
idxs = tuple(indexer)
nump_ans[idxs] = src.double().numpy()

print('ref numpy\n', ref.numpy())
print('idx numpy\n', idxs)
print('src numpy\n', src.numpy())

gpu_ans = ref.cuda()
gpu_ans[indexer]=src.cuda()

print('cpu result', cpu_ans)
print('numpy result\n', nump_ans)
print('cuda result', gpu_ans)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>

<span class="pl-s1">ref</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">arange</span>(<span class="pl-c1">0</span>, <span class="pl-c1">20</span>).<span class="pl-en">view</span>(<span class="pl-c1">4</span>, <span class="pl-c1">5</span>).<span class="pl-en">float</span>()

<span class="pl-s1">indexer</span><span class="pl-c1">=</span> [<span class="pl-en">slice</span>(<span class="pl-c1">None</span>), [<span class="pl-c1">1</span> <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">5</span>)]]

<span class="pl-s1">src</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">randperm</span>(<span class="pl-s1">ref</span>[<span class="pl-s1">indexer</span>].<span class="pl-en">numel</span>()).<span class="pl-en">view</span>(<span class="pl-s1">ref</span>[<span class="pl-s1">indexer</span>].<span class="pl-en">size</span>()).<span class="pl-en">float</span>()

<span class="pl-en">print</span>(<span class="pl-s">'ref'</span>, <span class="pl-s1">ref</span>)
<span class="pl-en">print</span>(<span class="pl-s">'indexer'</span>, <span class="pl-s1">indexer</span>)
<span class="pl-en">print</span>(<span class="pl-s">'ref[indxer]'</span>, <span class="pl-s1">ref</span>[<span class="pl-s1">indexer</span>])
<span class="pl-en">print</span>(<span class="pl-s">'src'</span>, <span class="pl-s1">src</span>)

<span class="pl-s1">cpu_ans</span> <span class="pl-c1">=</span> <span class="pl-s1">ref</span>.<span class="pl-en">clone</span>()
<span class="pl-s1">cpu_ans</span>[<span class="pl-s1">indexer</span>]<span class="pl-c1">=</span><span class="pl-s1">src</span>

<span class="pl-s1">nump_ans</span> <span class="pl-c1">=</span> <span class="pl-s1">ref</span>.<span class="pl-en">double</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">idxs</span> <span class="pl-c1">=</span> <span class="pl-en">tuple</span>(<span class="pl-s1">indexer</span>)
<span class="pl-s1">nump_ans</span>[<span class="pl-s1">idxs</span>] <span class="pl-c1">=</span> <span class="pl-s1">src</span>.<span class="pl-en">double</span>().<span class="pl-en">numpy</span>()

<span class="pl-en">print</span>(<span class="pl-s">'ref numpy<span class="pl-cce">\n</span>'</span>, <span class="pl-s1">ref</span>.<span class="pl-en">numpy</span>())
<span class="pl-en">print</span>(<span class="pl-s">'idx numpy<span class="pl-cce">\n</span>'</span>, <span class="pl-s1">idxs</span>)
<span class="pl-en">print</span>(<span class="pl-s">'src numpy<span class="pl-cce">\n</span>'</span>, <span class="pl-s1">src</span>.<span class="pl-en">numpy</span>())

<span class="pl-s1">gpu_ans</span> <span class="pl-c1">=</span> <span class="pl-s1">ref</span>.<span class="pl-en">cuda</span>()
<span class="pl-s1">gpu_ans</span>[<span class="pl-s1">indexer</span>]<span class="pl-c1">=</span><span class="pl-s1">src</span>.<span class="pl-en">cuda</span>()

<span class="pl-en">print</span>(<span class="pl-s">'cpu result'</span>, <span class="pl-s1">cpu_ans</span>)
<span class="pl-en">print</span>(<span class="pl-s">'numpy result<span class="pl-cce">\n</span>'</span>, <span class="pl-s1">nump_ans</span>)
<span class="pl-en">print</span>(<span class="pl-s">'cuda result'</span>, <span class="pl-s1">gpu_ans</span>)</pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cpu result 
  0   6   2   3   4
  5   4   7   8   9
 10  13  12  13  14
 15  19  17  18  19
[torch.FloatTensor of size 4x5]
numpy result
 [[  0.   6.   2.   3.   4.]
  [  5.   4.   7.   8.   9.]
  [ 10.  13.  12.  13.  14.]
  [ 15.  19.  17.  18.  19.]]
cuda result 
  0     6     2   3   4
  5  **16**   7   8   9
 10    13    12  13  14
 15   **5**  17  18  19
[torch.cuda.FloatTensor of size 4x5 (GPU 0)]"><pre class="notranslate"><code class="notranslate">cpu result 
  0   6   2   3   4
  5   4   7   8   9
 10  13  12  13  14
 15  19  17  18  19
[torch.FloatTensor of size 4x5]
numpy result
 [[  0.   6.   2.   3.   4.]
  [  5.   4.   7.   8.   9.]
  [ 10.  13.  12.  13.  14.]
  [ 15.  19.  17.  18.  19.]]
cuda result 
  0     6     2   3   4
  5  **16**   7   8   9
 10    13    12  13  14
 15   **5**  17  18  19
[torch.cuda.FloatTensor of size 4x5 (GPU 0)]
</code></pre></div>
<p dir="auto">Looks like  a preprocessing step will need to be added to identify which source indices is allowed to write to destination.</p>
<p dir="auto">I'm not sure if this will also be a problem in the other advanced indexing kernels.</p>
<p dir="auto"><strong>### Pretext:</strong></p>
<p dir="auto">I recently installed pytorch-nightly, and my cuda version is upto date. But when I run my command, I get the following error:</p>
<p dir="auto"><strong>### Error:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;main.py&quot;, line 77, in &lt;module&gt;
    algo.sample(datapath, cfg.STAGE)
  File &quot;C:\Users\hunte\OneDrive\Documents\Projects\EAD Project\StackGAN-Pytorch-master\code\trainer.py&quot;, line 238, in sample
    netG, _ = self.load_network_stageII()
  File &quot;C:\Users\hunte\OneDrive\Documents\Projects\EAD Project\StackGAN-Pytorch-master\code\trainer.py&quot;, line 110, in load_network_stageII
    netG.cuda()
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 263, in cuda
    return self._apply(lambda t: t.cuda(device))
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 190, in _apply
    module._apply(fn)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 190, in _apply
    module._apply(fn)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 190, in _apply
    module._apply(fn)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 196, in _apply
    param.data = fn(param.data)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py&quot;, line 263, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\torch\cuda\__init__.py&quot;, line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA error: unknown error"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "main.py", line 77, in &lt;module&gt;
    algo.sample(datapath, cfg.STAGE)
  File "C:\Users\hunte\OneDrive\Documents\Projects\EAD Project\StackGAN-Pytorch-master\code\trainer.py", line 238, in sample
    netG, _ = self.load_network_stageII()
  File "C:\Users\hunte\OneDrive\Documents\Projects\EAD Project\StackGAN-Pytorch-master\code\trainer.py", line 110, in load_network_stageII
    netG.cuda()
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 263, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 190, in _apply
    module._apply(fn)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 190, in _apply
    module._apply(fn)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 190, in _apply
    module._apply(fn)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 196, in _apply
    param.data = fn(param.data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 263, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\cuda\__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA error: unknown error
</code></pre></div>