<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: No. Built for iOS using provided scripts. Defined ops to be included in build as documented (using <code class="notranslate">ops_to_register.h</code>).</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Built on macOS 10.12.6 for distribution on iOS</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: source</li>
<li><strong>TensorFlow version (use command below)</strong>: Unsure, sorry, my python environment is all messed up right now. My HEAD is at <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/0d2f6918322c7bf29d1de3075b0d4ed3b1b72919/hovercard" href="https://github.com/tensorflow/tensorflow/commit/0d2f6918322c7bf29d1de3075b0d4ed3b1b72919"><tt>0d2f691</tt></a></li>
<li><strong>Python version</strong>: 2.7.12, but I believe that's irrelevant for this issue</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.5.1-homebrew, but I'm using <code class="notranslate">build_all_ios.sh</code> instead of Bazel to build</li>
<li><strong>CUDA/cuDNN version</strong>: Unknown</li>
<li><strong>GPU model and memory</strong>: iPhone 7 Plus, but AFAIK GPU is unavailable on iOS</li>
<li><strong>Exact command to reproduce</strong>: No particular command. Please see description of issue</li>
</ul>
<p dir="auto">Also <a href="https://stackoverflow.com/questions/45682837/tensorflow-thread-pools-appear-to-spin-forever-even-after-session-is-closed" rel="nofollow">asked about this</a> on Stack Overflow, where it was suggested I file an issue</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I believe this is a bug in Tensorflow. I am running Tensorflow on iOS, using the C++ API. I'm doing some image classification. I have a long-lived session, and I call <code class="notranslate">Run</code> many times on it, to evaluate different images from a backlog. Once I'm done, the <code class="notranslate">RunQueue</code>s (via <code class="notranslate">NonBlockingThreadPool</code>s) continue to pin the CPU at near max usage. They appear to be stuck in the <code class="notranslate">Steal</code> loop, presumably with no work to do.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">I tried <code class="notranslate">Close()</code>ing and then <code class="notranslate">delete</code>ing the session, and having read some of the C++ source, this <em>should</em> have shut down the thread pools that belong to the session, but this didn't change the situation:</p>
<div class="highlight highlight-source-objc++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="auto status = session-&gt;Close();
delete session;
session = nil;"><pre class="notranslate"><span class="pl-k">auto</span> status = session-&gt;<span class="pl-en">Close</span>();
<span class="pl-k">delete</span> session;
session = <span class="pl-c1">nil</span>;</pre></div>
<p dir="auto">I tried setting some specific configuration options so I could be sure that the session did in fact own its thread pools instead of using a global thread pool, but this didn't help either:</p>
<div class="highlight highlight-source-objc++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="tensorflow::SessionOptions options;
options.config.clear_session_inter_op_thread_pool();
options.config.set_use_per_session_threads(true);
auto status = tensorflow::NewSession(options, &amp;session);"><pre class="notranslate">tensorflow::SessionOptions options;
options.config.clear_session_inter_op_thread_pool();
options.config.set_use_per_session_threads(<span class="pl-c1">true</span>);
<span class="pl-k">auto</span> status = tensorflow::NewSession(options, &amp;session);</pre></div>
<p dir="auto">One thing to note: while my understanding is that this isn't necessary, I did also try using a mutex to ensure that <code class="notranslate">Close</code> and <code class="notranslate">delete</code> would not be called concurrently with any call to <code class="notranslate">Run</code>, but again, no luck.</p>
<p dir="auto">The only thing that <em>has</em> reduced CPU load to a reasonable level is to set <code class="notranslate">inter_op_parallelism_threads</code> to <code class="notranslate">1</code>. This doesn't resolve the underlying problem (that the threads are never cleaned up), but it <em>does</em> mean that the <code class="notranslate">Steal</code> loop is avoided, so the thread just blocks forever.</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Google Colab</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: N/A</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: N/A</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.10.0</li>
<li><strong>Python version</strong>: 3.6.3</li>
<li><strong>Bazel version (if compiling from source)</strong>: N/A</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: N/A</li>
<li><strong>CUDA/cuDNN version</strong>: N/A</li>
<li><strong>GPU model and memory</strong>: N/A</li>
<li><strong>Exact command to reproduce</strong>: N/A</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">When <code class="notranslate">train.batch_sequences_with_states</code> extracts batches from a sequence of samples, what it actually does is duplicating the same segment for <code class="notranslate">batch_size</code> times.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">The code is revised based on <a href="https://github.com/tensorflow/tensorflow/blob/r1.10/tensorflow/contrib/training/python/training/batch_sequences_with_states_test.py">batch_sequences_with_states_test.py</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from pprint import pprint

import numpy as np
import tensorflow as tf
from tensorflow.contrib.training.python.training import sequence_queueing_state_saver as sqss

batch_size = 3
num_unroll = 2
lstm_size = 4
value_length = 8
input_key = tf.as_string(tf.cast(10000 * tf.random_uniform(()), tf.int32))
input_sequences = {'input': np.random.rand(value_length, 3)}
input_context = {'context_key': [1]}
initial_states = {&quot;lstm_state&quot;: np.random.rand(10, lstm_size)}

with tf.Session() as sess:
    batch = sqss.batch_sequences_with_states(
        input_key=input_key,
        input_sequences=input_sequences,
        input_context=input_context,
        input_length=value_length,
        initial_states=initial_states,
        num_unroll=num_unroll,
        batch_size=batch_size)
    state = batch.state('lstm_state')
    update_state = batch.save_state('lstm_state', state + 1)
    coord = tf.train.Coordinator()
    tf.train.start_queue_runners(sess=sess, coord=coord)
    input_batch_val = sess.run([
        batch.key, batch.next_key, batch.sequences['input'],
        batch.context['context_key'], state, batch.length, update_state][2])
    pprint(input_batch_val)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">pprint</span> <span class="pl-k">import</span> <span class="pl-s1">pprint</span>

<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">from</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">training</span>.<span class="pl-s1">python</span>.<span class="pl-s1">training</span> <span class="pl-k">import</span> <span class="pl-s1">sequence_queueing_state_saver</span> <span class="pl-k">as</span> <span class="pl-s1">sqss</span>

<span class="pl-s1">batch_size</span> <span class="pl-c1">=</span> <span class="pl-c1">3</span>
<span class="pl-s1">num_unroll</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span>
<span class="pl-s1">lstm_size</span> <span class="pl-c1">=</span> <span class="pl-c1">4</span>
<span class="pl-s1">value_length</span> <span class="pl-c1">=</span> <span class="pl-c1">8</span>
<span class="pl-s1">input_key</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">as_string</span>(<span class="pl-s1">tf</span>.<span class="pl-en">cast</span>(<span class="pl-c1">10000</span> <span class="pl-c1">*</span> <span class="pl-s1">tf</span>.<span class="pl-en">random_uniform</span>(()), <span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>))
<span class="pl-s1">input_sequences</span> <span class="pl-c1">=</span> {<span class="pl-s">'input'</span>: <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-s1">value_length</span>, <span class="pl-c1">3</span>)}
<span class="pl-s1">input_context</span> <span class="pl-c1">=</span> {<span class="pl-s">'context_key'</span>: [<span class="pl-c1">1</span>]}
<span class="pl-s1">initial_states</span> <span class="pl-c1">=</span> {<span class="pl-s">"lstm_state"</span>: <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-c1">10</span>, <span class="pl-s1">lstm_size</span>)}

<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
    <span class="pl-s1">batch</span> <span class="pl-c1">=</span> <span class="pl-s1">sqss</span>.<span class="pl-en">batch_sequences_with_states</span>(
        <span class="pl-s1">input_key</span><span class="pl-c1">=</span><span class="pl-s1">input_key</span>,
        <span class="pl-s1">input_sequences</span><span class="pl-c1">=</span><span class="pl-s1">input_sequences</span>,
        <span class="pl-s1">input_context</span><span class="pl-c1">=</span><span class="pl-s1">input_context</span>,
        <span class="pl-s1">input_length</span><span class="pl-c1">=</span><span class="pl-s1">value_length</span>,
        <span class="pl-s1">initial_states</span><span class="pl-c1">=</span><span class="pl-s1">initial_states</span>,
        <span class="pl-s1">num_unroll</span><span class="pl-c1">=</span><span class="pl-s1">num_unroll</span>,
        <span class="pl-s1">batch_size</span><span class="pl-c1">=</span><span class="pl-s1">batch_size</span>)
    <span class="pl-s1">state</span> <span class="pl-c1">=</span> <span class="pl-s1">batch</span>.<span class="pl-en">state</span>(<span class="pl-s">'lstm_state'</span>)
    <span class="pl-s1">update_state</span> <span class="pl-c1">=</span> <span class="pl-s1">batch</span>.<span class="pl-en">save_state</span>(<span class="pl-s">'lstm_state'</span>, <span class="pl-s1">state</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>)
    <span class="pl-s1">coord</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">Coordinator</span>()
    <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-en">start_queue_runners</span>(<span class="pl-s1">sess</span><span class="pl-c1">=</span><span class="pl-s1">sess</span>, <span class="pl-s1">coord</span><span class="pl-c1">=</span><span class="pl-s1">coord</span>)
    <span class="pl-s1">input_batch_val</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>([
        <span class="pl-s1">batch</span>.<span class="pl-s1">key</span>, <span class="pl-s1">batch</span>.<span class="pl-s1">next_key</span>, <span class="pl-s1">batch</span>.<span class="pl-s1">sequences</span>[<span class="pl-s">'input'</span>],
        <span class="pl-s1">batch</span>.<span class="pl-s1">context</span>[<span class="pl-s">'context_key'</span>], <span class="pl-s1">state</span>, <span class="pl-s1">batch</span>.<span class="pl-s1">length</span>, <span class="pl-s1">update_state</span>][<span class="pl-c1">2</span>])
    <span class="pl-en">pprint</span>(<span class="pl-s1">input_batch_val</span>)</pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="array([[[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]],

       [[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]],

       [[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]]])"><pre class="notranslate"><code class="notranslate">array([[[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]],

       [[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]],

       [[0.33537843, 0.76504494, 0.368679  ],
        [0.47943187, 0.58871135, 0.06263617]]])
</code></pre></div>
<p dir="auto">What further justifies my suspect is the expected values of sequences in unit tests <a href="https://github.com/tensorflow/tensorflow/blob/r1.10/tensorflow/contrib/training/python/training/batch_sequences_with_states_test.py">batch_sequences_with_states_test.py</a>, which is duplicating the first segment of <code class="notranslate">"seq1"</code>.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def _testBasicPadding(self, pad, key=None, make_keys_unique=False):
    num_unroll = 2  # Divisor of value_length - so no padding necessary.
    expected_seq1_batch1 = np.tile(
        self.sequences[&quot;seq1&quot;][np.newaxis, 0:num_unroll, :],
(self.batch_size, 1, 1))"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">_testBasicPadding</span>(<span class="pl-s1">self</span>, <span class="pl-s1">pad</span>, <span class="pl-s1">key</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">make_keys_unique</span><span class="pl-c1">=</span><span class="pl-c1">False</span>):
    <span class="pl-s1">num_unroll</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span>  <span class="pl-c"># Divisor of value_length - so no padding necessary.</span>
    <span class="pl-s1">expected_seq1_batch1</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">tile</span>(
        <span class="pl-s1">self</span>.<span class="pl-s1">sequences</span>[<span class="pl-s">"seq1"</span>][<span class="pl-s1">np</span>.<span class="pl-s1">newaxis</span>, <span class="pl-c1">0</span>:<span class="pl-s1">num_unroll</span>, :],
(<span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>))</pre></div>