<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">When  use gpu to train a two-convolution-layer model, the random behavior is different in the running of two times.</p>
<h2 dir="auto">To Reproduce</h2>
<ol dir="auto">
<li>write this code in some file, on my computer the source code is named <code class="notranslate">try.py</code></li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import numpy as np
import random

torch.manual_seed(123)
random.seed(123)
np.random.seed(123)
torch.cuda.manual_seed(123)
torch.backends.cudnn.enabled=False
torch.backends.cudnn.deterministic=True

conv1 = torch.nn.Conv2d(3, 32, 3, 1, 1).cuda()
conv2 = torch.nn.Conv2d(32, 32, 3, stride=1, padding=1, dilation=1).cuda()
#  fc = torch.nn.Linear(32, 10)
criteria = torch.nn.CrossEntropyLoss()

torch.nn.init.kaiming_normal_(conv1.weight, 1)
torch.nn.init.kaiming_normal_(conv2.weight, 1)
params = list(conv1.parameters()) + list(conv2.parameters())
optim = torch.optim.SGD(params, lr=1e-4, momentum=0.9, weight_decay=1e-4)

for i in range(10):
    inten = torch.randn(16, 3, 64, 64).cuda()
    lb = torch.randint(0, 10, (16, 64, 64)).cuda()

    optim.zero_grad()
    feat = conv1(inten)
    feat = conv2(feat)
    loss = criteria(feat, lb)
    loss.backward()
    optim.step()
    print(loss.item())"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">random</span>

<span class="pl-s1">torch</span>.<span class="pl-en">manual_seed</span>(<span class="pl-c1">123</span>)
<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">123</span>)
<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">123</span>)
<span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-en">manual_seed</span>(<span class="pl-c1">123</span>)
<span class="pl-s1">torch</span>.<span class="pl-s1">backends</span>.<span class="pl-s1">cudnn</span>.<span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-c1">False</span>
<span class="pl-s1">torch</span>.<span class="pl-s1">backends</span>.<span class="pl-s1">cudnn</span>.<span class="pl-s1">deterministic</span><span class="pl-c1">=</span><span class="pl-c1">True</span>

<span class="pl-s1">conv1</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">3</span>, <span class="pl-c1">32</span>, <span class="pl-c1">3</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>).<span class="pl-en">cuda</span>()
<span class="pl-s1">conv2</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">32</span>, <span class="pl-c1">32</span>, <span class="pl-c1">3</span>, <span class="pl-s1">stride</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">dilation</span><span class="pl-c1">=</span><span class="pl-c1">1</span>).<span class="pl-en">cuda</span>()
<span class="pl-c">#  fc = torch.nn.Linear(32, 10)</span>
<span class="pl-s1">criteria</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">CrossEntropyLoss</span>()

<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">init</span>.<span class="pl-en">kaiming_normal_</span>(<span class="pl-s1">conv1</span>.<span class="pl-s1">weight</span>, <span class="pl-c1">1</span>)
<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">init</span>.<span class="pl-en">kaiming_normal_</span>(<span class="pl-s1">conv2</span>.<span class="pl-s1">weight</span>, <span class="pl-c1">1</span>)
<span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">conv1</span>.<span class="pl-en">parameters</span>()) <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">conv2</span>.<span class="pl-en">parameters</span>())
<span class="pl-s1">optim</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-v">SGD</span>(<span class="pl-s1">params</span>, <span class="pl-s1">lr</span><span class="pl-c1">=</span><span class="pl-c1">1e-4</span>, <span class="pl-s1">momentum</span><span class="pl-c1">=</span><span class="pl-c1">0.9</span>, <span class="pl-s1">weight_decay</span><span class="pl-c1">=</span><span class="pl-c1">1e-4</span>)

<span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">10</span>):
    <span class="pl-s1">inten</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-c1">16</span>, <span class="pl-c1">3</span>, <span class="pl-c1">64</span>, <span class="pl-c1">64</span>).<span class="pl-en">cuda</span>()
    <span class="pl-s1">lb</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-c1">10</span>, (<span class="pl-c1">16</span>, <span class="pl-c1">64</span>, <span class="pl-c1">64</span>)).<span class="pl-en">cuda</span>()

    <span class="pl-s1">optim</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-s1">feat</span> <span class="pl-c1">=</span> <span class="pl-en">conv1</span>(<span class="pl-s1">inten</span>)
    <span class="pl-s1">feat</span> <span class="pl-c1">=</span> <span class="pl-en">conv2</span>(<span class="pl-s1">feat</span>)
    <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">criteria</span>(<span class="pl-s1">feat</span>, <span class="pl-s1">lb</span>)
    <span class="pl-s1">loss</span>.<span class="pl-en">backward</span>()
    <span class="pl-s1">optim</span>.<span class="pl-en">step</span>()
    <span class="pl-en">print</span>(<span class="pl-s1">loss</span>.<span class="pl-en">item</span>())</pre></div>
<ol start="2" dir="auto">
<li>run the above code for twice, and see the loss printed to the screen:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101936340332
3.944667100906372
3.9629740715026855
3.9574058055877686
3.9502787590026855
3.956238031387329
3.9543356895446777
3.950388193130493
3.946221351623535

CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101697921753
3.944666862487793
3.9629745483398438
3.9574055671691895
3.9502787590026855
3.956238031387329
3.9543352127075195
3.950387954711914
3.9462215900421143"><pre class="notranslate"><code class="notranslate">CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101936340332
3.944667100906372
3.9629740715026855
3.9574058055877686
3.9502787590026855
3.956238031387329
3.9543356895446777
3.950388193130493
3.946221351623535

CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101697921753
3.944666862487793
3.9629745483398438
3.9574055671691895
3.9502787590026855
3.956238031387329
3.9543352127075195
3.950387954711914
3.9462215900421143
</code></pre></div>
<p dir="auto">Since my practical model has a big variance, this random behavior can bring a final score variance of as much as 0.5%, which makes it too difficult for me to tune the hyper-parameters. What is the cause of this randomness, and how could I reduce it please?</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Collecting environment information...<br>
PyTorch version: 1.1.0a0+95ce796<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p dir="auto">OS: CentOS Linux 7 (Core)<br>
GCC version: (GCC) 5.4.0<br>
CMake version: version 3.14.0</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.0.176<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GTX 1080 Ti<br>
GPU 3: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 390.67<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.16.3<br>
[pip] torch==1.1.0<br>
[pip] torchvision==0.2.2.post3<br>
[pip] torchvision-nightly==0.2.3<br>
[conda] Could not collect</p>
<p dir="auto">Hi,</p>
<p dir="auto">I came across this problem while using pytorch:</p>
<p dir="auto">The dot product between 2 vectors with floating precision can become arbitrarily small/large. This problem does not occur when using the sum and product operation separately:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v1, v2 = torch.FloatTensor([1., 1.]), torch.FloatTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))  
(-3.6893488147419103e+19, -1.9999998807907104)"><pre class="notranslate"><code class="notranslate">v1, v2 = torch.FloatTensor([1., 1.]), torch.FloatTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))  
(-3.6893488147419103e+19, -1.9999998807907104)
</code></pre></div>
<p dir="auto">Dot product with tensors with double precision seems ok:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v1, v2 = torch.DoubleTensor([1., 1.]), torch.DoubleTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))
(-1.9999999000000002, -1.9999999000000002)"><pre class="notranslate"><code class="notranslate">v1, v2 = torch.DoubleTensor([1., 1.]), torch.DoubleTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))
(-1.9999999000000002, -1.9999999000000002)
</code></pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: None</p>
<p dir="auto">OS: Mac OSX 10.13.4<br>
GCC version: Could not collect<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: No<br>
CUDA runtime version: No CUDA<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy (1.14.0)<br>
[pip] numpydoc (0.7.0)<br>
[pip] torch (0.4.0)<br>
[pip] torchvision (0.2.1)<br>
[conda] pytorch                   0.4.0           py36_cuda0.0_cudnn0.0_1    pytorch<br>
[conda] torchvision               0.2.1                    py36_1    pytorch</p>