<p dir="auto">Feature request: The <em>sequence_length</em> parameter in rnn.rnn() is a convenient way to support variable length sequences. The great thing about it is that it propagates the state at an example's sequence length to the final state output. However, it doesn't similarly propagate the output at an example's sequence length. This is the feature being requested here.</p>
<p dir="auto">If one is interested in the last output only, reading outputs[-1] returns zeros for all the examples that are shorter than the max sequence length. It would simplify things considerably if there was a way to get the output value at the example's sequence length.</p>
<p dir="auto">So I'm trying to figure out why my resnets are running out of memory, and it seems that there's a memory leak in Tile and zeros_like operations.</p>
<p dir="auto">Those ops have memory allocated during each session run but there's no <code class="notranslate">__LOG_MEMORY__</code> deallocation messages corresponding to them. The sum of missing deallocations matches the amount of memory leaked as reported by allocator as <code class="notranslate">max_bytes_in_use</code> (accessed through <code class="notranslate">tf.contrib.memory_stats.MaxBytesInUse</code> op)</p>
<p dir="auto">Here's a simplified repro, at each sess.run, the memory grows by 1.15 GB until it crashes with OOM<br>
<a href="https://github.com/yaroslavvb/stuff/blob/master/resnet_leak_report2.py">https://github.com/yaroslavvb/stuff/blob/master/resnet_leak_report2.py</a></p>
<p dir="auto">When I run it, I see</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Run 0, GBs in use 2.30
Run 1, GBs in use 3.60
Run 2, GBs in use 4.75
Run 3, GBs in use 5.90
2017-09-21 14:56:31.994302: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 137.33MiB.  Current allocation summary follows...."><pre class="notranslate"><code class="notranslate">Run 0, GBs in use 2.30
Run 1, GBs in use 3.60
Run 2, GBs in use 4.75
Run 3, GBs in use 5.90
2017-09-21 14:56:31.994302: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 137.33MiB.  Current allocation summary follows....
</code></pre></div>
<p dir="auto">Offending ops:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gradients/leaky_relu_grad/zeros_like 576MB
gradients/Sum_grad/Tile  576MB"><pre class="notranslate"><code class="notranslate">gradients/leaky_relu_grad/zeros_like 576MB
gradients/Sum_grad/Tile  576MB
</code></pre></div>
<p dir="auto">Version:<br>
Ubuntu 16:04<br>
official TensorFlow Linux GPU Python 3.5 nightly wheel from <a href="https://ci.tensorflow.org/view/tf-nightly/job/tf-nightly-linux/TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3.5,label=gpu-linux/lastSuccessfulBuild/artifact/pip_test/whl/tf_nightly_gpu-1.head-cp35-cp35m-linux_x86_64.whl" rel="nofollow">today</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="version: 1.4.0-dev20170921
__git_version__: v1.3.0-rc1-2408-ge9d5ee1
Commit https://github.com/tensorflow/tensorflow/commit/e9d5ee1
"><pre class="notranslate"><code class="notranslate">version: 1.4.0-dev20170921
__git_version__: v1.3.0-rc1-2408-ge9d5ee1
Commit https://github.com/tensorflow/tensorflow/commit/e9d5ee1

</code></pre></div>