<p dir="auto">Hello!<br>
I have been working my way through the  tf.contrib.learn <a href="https://www.tensorflow.org/tutorials/" rel="nofollow">tutorials</a> and have been attempting to integrate the tf.contrib.learn.monitors.ValidationMonitor into the 'deep' classifier in wide_n_deep.py as shown below.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?  I searched both Github and Stakeoverflow with the terms 'tensorflow,' 'input_fn,' and 'validationmonitor' but wasn't able to find anyone else who reported similar issues.</h3>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:  I ran this on Ubuntu Server 16.04 on a physical I7 with a GTX1080 gpu when i noticed the problem.  I know that i was using the GPU on the original physical box from previous tests, and because during the hang the nvidia_smi command showed considerable load on the GPU.  I was able to replicate the problem with CPU on a 16.04 VM as well.</p>
<p dir="auto">Installed version of CUDA and cuDNN:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/home/andersonjas/libcudnn5-dev_5.1.5-1+cuda8.0_amd64.deb
/home/andersonjas/libcudnn5_5.1.5-1+cuda8.0_amd64.deb"><pre class="notranslate"><code class="notranslate">/home/andersonjas/libcudnn5-dev_5.1.5-1+cuda8.0_amd64.deb
/home/andersonjas/libcudnn5_5.1.5-1+cuda8.0_amd64.deb
</code></pre></div>
<p dir="auto">If installed from binary pip package, provide:</p>
<ol dir="auto">
<li>A link to the pip package you installed:<br>
from Anaconda 2.7 64 bit package:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install tensorflow"><pre class="notranslate"><code class="notranslate">pip install tensorflow
</code></pre></div>
<ol start="2" dir="auto">
<li>The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="andersonjas@ubuntu:~$ python -c &quot;import tensorflow; print(tensorflow.__version__)&quot;
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
0.11.head
"><pre class="notranslate"><code class="notranslate">andersonjas@ubuntu:~$ python -c "import tensorflow; print(tensorflow.__version__)"
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
0.11.head

</code></pre></div>
<p dir="auto">If installed from source, provide</p>
<ol dir="auto">
<li>The commit hash (<code class="notranslate">git rev-parse HEAD</code>)</li>
<li>The output of <code class="notranslate">bazel version</code></li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="validation_monitor = tf.contrib.learn.monitors.ValidationMonitor(input_fn=lambda:input_fn(df_test), 
                       every_n_steps=50)
m.fit(input_fn=lambda: input_fn(df_train), steps=151,monitors=[validation_monitor])"><pre class="notranslate"><span class="pl-s1">validation_monitor</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">learn</span>.<span class="pl-s1">monitors</span>.<span class="pl-v">ValidationMonitor</span>(<span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>:<span class="pl-en">input_fn</span>(<span class="pl-s1">df_test</span>), 
                       <span class="pl-s1">every_n_steps</span><span class="pl-c1">=</span><span class="pl-c1">50</span>)
<span class="pl-s1">m</span>.<span class="pl-en">fit</span>(<span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>: <span class="pl-en">input_fn</span>(<span class="pl-s1">df_train</span>), <span class="pl-s1">steps</span><span class="pl-c1">=</span><span class="pl-c1">151</span>,<span class="pl-s1">monitors</span><span class="pl-c1">=</span>[<span class="pl-s1">validation_monitor</span>])</pre></div>
<p dir="auto">Doing this in a jupyter notebool causes the code to hang indefinitely.  To make completely sure that i don't have a bug in my own code i can make the following change:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="validation_monitor = tf.contrib.learn.monitors.ValidationMonitor(input_fn=lambda:input_fn(df_test), 
                       every_n_steps=50)
m.fit(input_fn=lambda: input_fn(df_train), steps=151) #,monitors=[validation_monitor])"><pre class="notranslate"><span class="pl-s1">validation_monitor</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">learn</span>.<span class="pl-s1">monitors</span>.<span class="pl-v">ValidationMonitor</span>(<span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>:<span class="pl-en">input_fn</span>(<span class="pl-s1">df_test</span>), 
                       <span class="pl-s1">every_n_steps</span><span class="pl-c1">=</span><span class="pl-c1">50</span>)
<span class="pl-s1">m</span>.<span class="pl-en">fit</span>(<span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>: <span class="pl-en">input_fn</span>(<span class="pl-s1">df_train</span>), <span class="pl-s1">steps</span><span class="pl-c1">=</span><span class="pl-c1">151</span>) <span class="pl-c">#,monitors=[validation_monitor])</span></pre></div>
<p dir="auto">and then the code executes fine.</p>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<p dir="auto">I also built an input_fn interface to the iris and boston housing price predictor code code, each showed similar 'hangs'</p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>
<p dir="auto">As a noob, i'm learning that esoteric error messages are a luxury :-),  in this case the code just hangs indefinitely.</p>
<p dir="auto">hi all,</p>
<p dir="auto">it would be really great to have a tutorial on how to use the many awesome wrappers like dropoutwrapper, attentionwrapper, etc.</p>