<p dir="auto"><em>Please make sure that this is a feature request. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>TensorFlow version (you are using): 2.0</li>
<li>Are you willing to contribute it (Yes/No): Happy to help as much as I can!</li>
</ul>
<p dir="auto"><strong>Describe the feature and the current behavior/state.</strong><br>
TensorFlow 1.x support specifying GPU devices to use:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Horovod: pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.visible_device_list = str(hvd.local_rank())"><pre class="notranslate"><span class="pl-c"># Horovod: pin GPU to be used to process local rank (one GPU per process)</span>
<span class="pl-s1">config</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">ConfigProto</span>()
<span class="pl-s1">config</span>.<span class="pl-s1">gpu_options</span>.<span class="pl-s1">visible_device_list</span> <span class="pl-c1">=</span> <span class="pl-en">str</span>(<span class="pl-s1">hvd</span>.<span class="pl-en">local_rank</span>())</pre></div>
<p dir="auto">There's no comparable API in TensorFlow 2.0.  The closest option is to use the <code class="notranslate">CUDA_VISIBLE_DEVICES</code> environment variable.  Unfortunately, <code class="notranslate">CUDA_VISIBLE_DEVICES</code> prevents processes from doing <code class="notranslate">cudaMemcpy</code> from/to devices not owned by the process.  There's a significant performance degradation when NCCL is used with P2P communication disabled.</p>
<p dir="auto">The ask is to add an API to TensorFlow 2.0 to enable device selection.</p>
<p dir="auto"><strong>Will this change the current api? How?</strong><br>
Yes, will introduce an API to select GPU devices to use.</p>
<p dir="auto"><strong>Who will benefit with this feature?</strong><br>
Users of <a href="http://horovod.ai" rel="nofollow">Horovod</a>.</p>
<p dir="auto"><strong>Any Other info.</strong><br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/azaks2/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/azaks2">@azaks2</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/alextp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/alextp">@alextp</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jaingaurav/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jaingaurav">@jaingaurav</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/guptapriya/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/guptapriya">@guptapriya</a></p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li>
<p dir="auto"><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:<br>
Yes</p>
</li>
<li>
<p dir="auto"><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
Linux Ubuntu 16.04</p>
</li>
<li>
<p dir="auto"><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>:</p>
</li>
<li>
<p dir="auto"><strong>TensorFlow installed from (source or binary)</strong>:<br>
My own build from source as well as google binary found at: <a href="https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.8.0.tar.gz" rel="nofollow">https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.8.0.tar.gz</a></p>
</li>
<li>
<p dir="auto"><strong>TensorFlow version (use command below)</strong>:<br>
1.8.0 - using C APi, not python</p>
</li>
<li>
<p dir="auto"><strong>Python version</strong>:</p>
</li>
<li>
<p dir="auto"><strong>Bazel version (if compiling from source)</strong>:</p>
</li>
<li>
<p dir="auto"><strong>GCC/Compiler version (if compiling from source)</strong>:</p>
</li>
<li>
<p dir="auto"><strong>CUDA/cuDNN version</strong>:</p>
</li>
<li>
<p dir="auto"><strong>GPU model and memory</strong>:</p>
</li>
<li>
<p dir="auto"><strong>Exact command to reproduce</strong>:</p>
</li>
</ul>
<p dir="auto">N/A, using google TF 1.8.0 cpu build, calling libtensorflow.so via C API</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I get valgrind reported memory leak(s), when using TF 1.8 (updating from previously used TF 1.2).  Specifically, the leak comes from nsync::nsync_waiter_new_().   The valgrind output looks like this:</p>
<p dir="auto"><code class="notranslate">==1947== 400 bytes in 1 blocks are possibly lost in loss record 80,541 of 82,598 ==1947==    at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==1947==    by 0xC9A6C34: nsync::nsync_waiter_new_() (in /usr/local/lib/libtensorflow_framework.so) ==1947==    by 0xC9A741E: nsync::nsync_cv_wait_with_deadline_generic(nsync::nsync_cv_s_*, void*, void (*)(void*), void (*)(void*), timespec, nsync::nsync_note_s_*) (in /usr/local/lib/libtensorflow_framework.so) ==1947==    by 0xC9A7AE4: nsync::nsync_cv_wait_with_deadline(nsync::nsync_cv_s_*, nsync::nsync_mu_s_*, timespec, nsync::nsync_note_s_*) (in /usr/local/lib/libtensorflow_framework.so) ==1947==    by 0x73DCA4A: ??? (in /usr/local/lib/libtensorflow.so) ==1947==    by 0x73DCA9A: ??? (in /usr/local/lib/libtensorflow.so) ==1947==    by 0x73E228F: ??? (in /usr/local/lib/libtensorflow.so) ==1947==    by 0x73EBCE4: ??? (in /usr/local/lib/libtensorflow.so) ==1947==    by 0x5258D29: ??? (in /usr/local/lib/libtensorflow.so) ==1947==    by 0x5259925: TF_SessionRun (in /usr/local/lib/libtensorflow.so)</code></p>
<p dir="auto">I think I may have found the source by looking at the nsync code located at: <a href="https://github.com/google/nsync/blob/master/internal/common.c">https://github.com/google/nsync/blob/master/internal/common.c</a></p>
<p dir="auto">I cannot observe where the memory allocated by malloc() in nsync_waiter_new_() is freed().  Furthermore, I don't see how the memory could safely be freed because the function allows for a caller-supplied allocator, which it conditionally uses, but does not seem to record which allocator was used, or have a mechanism for calling a caller-provided free().</p>
<p dir="auto">I believe that this is a memory leak bug in Tensorflow, being caused by the nsync dependency.</p>
<p dir="auto">Because the optimized monolithic build of libtensorflow does not seem to have debug symbols enabled by default, and because this call can occur in so many different contexts; it is difficult to generate an appropriate valgrind suppresions file which does not suppress too much.</p>