<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Training with nn.DataParallel in v1.1(v1.2 or v1.3) is much slower than v1.0.1.</p>
<p dir="auto">Such as run_squad.py in transformers library as its README<br>
<a href="url">https://github.com/huggingface/transformers/blob/master/examples/run_squad.py</a></p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">export SQUAD_DIR=/path/to/SQUAD</p>
<p dir="auto">python run_squad.py <br>
--model_type bert <br>
--model_name_or_path bert-base-cased <br>
--do_train <br>
--do_eval <br>
--do_lower_case <br>
--train_file $SQUAD_DIR/train-v1.1.json <br>
--predict_file $SQUAD_DIR/dev-v1.1.json <br>
--per_gpu_train_batch_size 5 <br>
--learning_rate 3e-5 <br>
--num_train_epochs 2.0 <br>
--max_seq_length 384 <br>
--doc_stride 128 <br>
--output_dir /tmp/debug_squad/</p>
<p dir="auto">The smaller batch size the speed is more distinct.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">They should have almost the same speed.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">For v1.0.1<br>
PyTorch version: 1.0.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
CMake version: version 3.10.2</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GTX 1080 Ti<br>
GPU 3: GeForce GTX 1080 Ti<br>
GPU 4: GeForce GTX 1080 Ti<br>
GPU 5: GeForce GTX 1080 Ti<br>
GPU 6: GeForce GTX 1080 Ti<br>
GPU 7: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 410.104<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.16.2<br>
[pip] numpydoc==0.9.1<br>
[pip] torch==1.0.1<br>
[conda] blas                      1.0                         mkl    defaults<br>
[conda] mkl                       2019.4                      243    defaults<br>
[conda] mkl-service               2.0.2            py37h7b6447c_0    defaults<br>
[conda] mkl_fft                   1.0.14           py37ha843d7b_0    defaults<br>
[conda] mkl_random                1.0.2            py37hd81dba3_0    defaults<br>
[conda] pytorch                   1.0.1           cuda100py37he554f03_0    defaults</p>
<p dir="auto">For v1.3<br>
PyTorch version: 1.3.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
CMake version: version 3.10.2</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GTX 1080 Ti<br>
GPU 3: GeForce GTX 1080 Ti<br>
GPU 4: GeForce GTX 1080 Ti<br>
GPU 5: GeForce GTX 1080 Ti<br>
GPU 6: GeForce GTX 1080 Ti<br>
GPU 7: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 410.104<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.16.2<br>
[pip] numpydoc==0.9.1<br>
[pip] torch==1.3.0<br>
[conda] blas                      1.0                         mkl    defaults<br>
[conda] mkl                       2019.4                      243    defaults<br>
[conda] mkl-service               2.3.0            py37he904b0f_0    defaults<br>
[conda] mkl_fft                   1.0.14           py37ha843d7b_0    defaults<br>
[conda] mkl_random                1.1.0            py37hd6b4f25_0    defaults<br>
[conda] pytorch                   1.3.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch</p>
<p dir="auto">Hi,<br>
I'm working with Pytorch 0.4, I got the  FileNotFoundError Exception in functional.py when it invokes the binary_cross_entropy loss. Here is the stack trace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="loss += torch.mean(F.binary_cross_entropy(F.sigmoid(out0), all0) +"><pre class="notranslate"><code class="notranslate">loss += torch.mean(F.binary_cross_entropy(F.sigmoid(out0), all0) +
</code></pre></div>
<p dir="auto">File "/home/local/.../anaconda2/envs/py27/lib/python2.7/site-packages/torch/nn/functional.py", line 1483, in binary_cross_entropy<br>
return torch._C._nn.binary_cross_entropy(input, target, weight, size_average, reduce)<br>
RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered</p>
<p dir="auto">I'm aware of this type of issues like <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="317476438" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/6932" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/6932/hovercard" href="https://github.com/pytorch/pytorch/issues/6932">#6932</a>. I added the fix in the locations suggested by the users in threads, however, the exception still occurs.  I'd be happy for any help.<br>
Best,<br>
S.</p>