<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): <a href="https://github.com/tensorflow/benchmarks/tree/cnn_tf_v1.12_compatible">tensorflow/benchmarks:cnn_tf_v1.12_compatible</a></li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04-hwe</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): ('v1.12.0-0-ga6d8ffae09', '1.12.0')</li>
<li>Python version: 2.7.12</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: 9.0/7.2.1.38-1+cuda9.0</li>
<li>GPU model and memory: V100-PCIE-32GB, NVIDIA driver version 384.145, with <code class="notranslate">intel_iommu=on</code></li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=/dev/mapper/vgroot-lvroot ro intel_iommu=on"><pre class="notranslate"><code class="notranslate">$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=/dev/mapper/vgroot-lvroot ro intel_iommu=on
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	CPU Affinity
GPU0	 X 	PIX	NODE	NODE	0-9,20-29
GPU1	PIX	 X 	NODE	NODE	0-9,20-29
GPU2	NODE	NODE	 X 	PIX	0-9,20-29
GPU3	NODE	NODE	PIX	 X 	0-9,20-29

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks"><pre class="notranslate"><code class="notranslate">$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	CPU Affinity
GPU0	 X 	PIX	NODE	NODE	0-9,20-29
GPU1	PIX	 X 	NODE	NODE	0-9,20-29
GPU2	NODE	NODE	 X 	PIX	0-9,20-29
GPU3	NODE	NODE	PIX	 X 	0-9,20-29

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks
</code></pre></div>
<p dir="auto"><strong>Describe the current behavior</strong><br>
FP-16 multi-GPU training with CPU as local parameter server is converging in single process mode, but diverging loss value (nan) adding another loopback PS process with grpc. Consistent behaviours for both ResNet-50 and VGG16.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
The same training procedure should yield exactly same result using only 1 worker with/without 1 PS. Not sure why adding SendRecvOps causes data corruption. See other info below.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<p dir="auto">ResNet50 local with <a href="https://gist.github.com/byronyi/4f0821a4792b479bf884ff81171c4011">output</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=resnet50 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256"><pre class="notranslate"><code class="notranslate">$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=resnet50 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256
</code></pre></div>
<p dir="auto">VGG16 local with <a href="https://gist.github.com/byronyi/0dc379b8feac762bc8d04cab2b240060">output</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=vgg16 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256"><pre class="notranslate"><code class="notranslate">$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=vgg16 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256
</code></pre></div>
<p dir="auto">Distributed using the same PS command with <a href="https://gist.github.com/byronyi/ea508b150d1e55b0ac41502c48685b7c">output</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CUDA_VISIBLE_DEVICES= \
python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=ps \
  --local_parameter_device=cpu \
  --task_index=0 \
  --server_protocol=grpc"><pre class="notranslate"><code class="notranslate">CUDA_VISIBLE_DEVICES= \
python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=ps \
  --local_parameter_device=cpu \
  --task_index=0 \
  --server_protocol=grpc
</code></pre></div>
<p dir="auto">ResNet50 distributed with <a href="https://gist.github.com/byronyi/de05ccb7b481932ce7be91f889f41aee">output</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=worker \
  --task_index=0 \
  --server_protocol=grpc \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=resnet50 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256"><pre class="notranslate"><code class="notranslate">$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=worker \
  --task_index=0 \
  --server_protocol=grpc \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=resnet50 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256
</code></pre></div>
<p dir="auto">VGG16 distributed with <a href="https://gist.github.com/byronyi/1113a5cd8048f729a2c73df5bfd97017">output</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=worker \
  --task_index=0 \
  --server_protocol=grpc \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=vgg16 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256"><pre class="notranslate"><code class="notranslate">$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
  --ps_hosts=localhost:5000 \
  --worker_hosts=localhost:5001 \
  --job_name=worker \
  --task_index=0 \
  --server_protocol=grpc \
  --variable_update=parameter_server \
  --local_parameter_device=cpu \
  --model=vgg16 \
  --num_gpus=4 \
  --use_fp16 \
  --batch_size=256
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
Kernel reports <code class="notranslate">[DMA Write] PTE Write access is not set</code> for one of the GPU.<br>
Detailed log could be found <a href="https://gist.github.com/dd1e75702116d941d95dfa1fe17b8e27">here</a>.</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: No</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04 LTS (GNU/Linux 4.4.38-rt49-tegra aarch64)</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: source</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.7.0-rc0, r1.7, master</li>
<li><strong>Python version</strong>: 2.7.12</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.11.1</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: 5.4.0-6ubuntu1~16.04.9</li>
<li><strong>CUDA/cuDNN version</strong>: 8.0/6</li>
<li><strong>GPU model and memory</strong>: name: GP106 major: 6 minor: 1 totalMemory: 3.75GiB (as reported by some previous version of Tensorflow on the other part of the same Drive PX2)</li>
<li><strong>Exact command to reproduce</strong>:<br>
<code class="notranslate">git checkout v1.7.0-rc0</code><br>
<code class="notranslate">./configure</code><br>
<code class="notranslate">bazel build //tensorflow:libtensorflow_cc.so</code></li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Same problem for all branches listed under <strong>TensorFlow version</strong>.<br>
Build fails immediately with<br>
<code class="notranslate">Cuda Configuration Error: Cannot find libdevice.10.bc under /usr/local/cuda-8.0</code><br>
It's strange because I explicitly answered 8.0 versions and libdevice.10.bc exists only in 9.0 AFAIK.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/1822159/error.log">error.log</a></p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/1822155/configure.log">configure.log</a></p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/1822157/tf_configure.txt">tf_configure.txt</a></p>