<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>

<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
a = torch.arange(1,17).float().reshape(4,4)
print(a)
cpu_res = torch.mean(a, (0,0))
cuda_res = torch.mean(a.cuda(), (0,0))
print(cpu_res)
print(cuda_res)"><pre class="notranslate"><code class="notranslate">import torch
a = torch.arange(1,17).float().reshape(4,4)
print(a)
cpu_res = torch.mean(a, (0,0))
cuda_res = torch.mean(a.cuda(), (0,0))
print(cpu_res)
print(cuda_res)
</code></pre></div>
<p dir="auto">Result is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tensor([[ 1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.],
        [ 9., 10., 11., 12.],
        [13., 14., 15., 16.]])
tensor([1.7500, 2.0000, 2.2500, 2.5000])
tensor([ 7.,  8.,  9., 10.], device='cuda:0')"><pre class="notranslate"><code class="notranslate">tensor([[ 1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.],
        [ 9., 10., 11., 12.],
        [13., 14., 15., 16.]])
tensor([1.7500, 2.0000, 2.2500, 2.5000])
tensor([ 7.,  8.,  9., 10.], device='cuda:0')
</code></pre></div>
<p dir="auto">When transfer a same dim like 0 twice, CUDA and CPU return different result. It seems like CPU calculates mean twice and CUDA not.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">I am not sure if this is reproducible for every environment, but I hit the following error when trying to set cuda device in processes. What is weird is that the error disappears if I remove the line <code class="notranslate">x = torch.rand(20, 2).cuda()</code> right after the for loop.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py&quot;, line 297, in _bootstrap
    self.run()
  File &quot;/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py&quot;, line 99, in run
    self._target(*self._args, **self._kwargs)
  File &quot;dist_bug.py&quot;, line 5, in run
    torch.cuda.set_device(rank)
  File &quot;/home/shenli/project/pytorch/torch/cuda/__init__.py&quot;, line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (3) : initialization error at ../torch/csrc/cuda/Module.cpp:33"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "dist_bug.py", line 5, in run
    torch.cuda.set_device(rank)
  File "/home/shenli/project/pytorch/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (3) : initialization error at ../torch/csrc/cuda/Module.cpp:33
</code></pre></div>
<p dir="auto"><strong>Edit:</strong>  based on the discussion below, the solution should be fixing the "bad fork" error detection, which is duplicated with <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="413035669" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/17359" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/17359/hovercard" href="https://github.com/pytorch/pytorch/issues/17359">#17359</a>.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
from torch.multiprocessing import Process

def run(rank):
    torch.cuda.set_device(rank)

if __name__ == &quot;__main__&quot;:
    size = 2
    processes = []
    for rank in range(size):
        # it would work fine without the line below
        x = torch.rand(20, 2).cuda()
        p = Process(target=run, args=(rank,))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">multiprocessing</span> <span class="pl-k">import</span> <span class="pl-v">Process</span>

<span class="pl-k">def</span> <span class="pl-en">run</span>(<span class="pl-s1">rank</span>):
    <span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-en">set_device</span>(<span class="pl-s1">rank</span>)

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-s1">size</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span>
    <span class="pl-s1">processes</span> <span class="pl-c1">=</span> []
    <span class="pl-k">for</span> <span class="pl-s1">rank</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">size</span>):
        <span class="pl-c"># it would work fine without the line below</span>
        <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">20</span>, <span class="pl-c1">2</span>).<span class="pl-en">cuda</span>()
        <span class="pl-s1">p</span> <span class="pl-c1">=</span> <span class="pl-v">Process</span>(<span class="pl-s1">target</span><span class="pl-c1">=</span><span class="pl-s1">run</span>, <span class="pl-s1">args</span><span class="pl-c1">=</span>(<span class="pl-s1">rank</span>,))
        <span class="pl-s1">p</span>.<span class="pl-en">start</span>()
        <span class="pl-s1">processes</span>.<span class="pl-en">append</span>(<span class="pl-s1">p</span>)

    <span class="pl-k">for</span> <span class="pl-s1">p</span> <span class="pl-c1">in</span> <span class="pl-s1">processes</span>:
        <span class="pl-s1">p</span>.<span class="pl-en">join</span>()</pre></div>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.1.0a0+63214b5<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.2.88</p>
<p dir="auto">OS: CentOS Linux 7 (Core)<br>
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)<br>
CMake version: version 3.12.2</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.2.88<br>
GPU models and configuration:<br>
GPU 0: Tesla M40<br>
GPU 1: Tesla M40</p>
<p dir="auto">Nvidia driver version: 396.26<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.15.4<br>
[pip] torch==1.1.0a0+63214b5<br>
[conda] blas 1.0 mkl<br>
[conda] mkl 2019.1 144<br>
[conda] mkl-include 2019.1 144<br>
[conda] mkl_fft 1.0.6 py37hd81dba3_0<br>
[conda] mkl_random 1.0.2 py37hd81dba3_0<br>
[conda] torch 1.1.0a0+63214b5 dev_0</p>