<p dir="auto">I've run into a strange problem involving both <code class="notranslate">tf.train.Saver</code> and an input pipeline based on <code class="notranslate">tf.train.string_input_producer()</code>. It seems as though the <code class="notranslate">num_epochs</code> parameter to <code class="notranslate">tf.train.string_input_producer()</code> stops working as intended after loading a saved model.</p>
<p dir="auto">I'm currently using v0.8 but I suspect this is still an issue on v0.9 and above.</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Mac OSX (10.11.6)<br>
Tensorflow version: 0.8<br>
pip version: 8.1.1</p>
<h3 dir="auto">Steps to reproduce</h3>
<p dir="auto">1 - Create a new directory and add to it the files from the following gist: <a href="https://gist.github.com/6d730e1b3d331c4be34c9a57b0a39ccf">https://gist.github.com/6d730e1b3d331c4be34c9a57b0a39ccf</a> . The <code class="notranslate">1.txt</code> and <code class="notranslate">2.txt</code> files simulate training data. The <code class="notranslate">training.py</code> file implements a simple input pipeline, taking in text files, and printing them line by line. Notice that the <code class="notranslate">num_epochs</code> parameter on line 28 is set to 1.</p>
<p dir="auto">2 - Run <code class="notranslate">$ python training.py test 1.txt</code>. The output should resemble the following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="The multiplying villanies of nature
Do swarm upon him--from the western isles
And fortune, on his damned quarrel smiling,
Show'd like a rebel's whore: but all's too weak:
For brave Macbeth--well he deserves that name--
Model saved to ./model-test-0
As two spent swimmers, that do cling together
Disdaining fortune, with his brandish'd steel,
And fix'd his head upon our battlements.
Like valour's minion carved out his passage
Which smoked with bloody execution,
Model saved to ./model-test-1
Till he unseam'd him from the nave to the chaps,
And choke their art. The merciless Macdonwald--
Of kerns and gallowglasses is supplied;
Worthy to be a rebel, for to that
Doubtful it stood;
Model saved to ./model-test-2
Done training!
Model saved to ./model-test-3"><pre class="notranslate"><code class="notranslate">The multiplying villanies of nature
Do swarm upon him--from the western isles
And fortune, on his damned quarrel smiling,
Show'd like a rebel's whore: but all's too weak:
For brave Macbeth--well he deserves that name--
Model saved to ./model-test-0
As two spent swimmers, that do cling together
Disdaining fortune, with his brandish'd steel,
And fix'd his head upon our battlements.
Like valour's minion carved out his passage
Which smoked with bloody execution,
Model saved to ./model-test-1
Till he unseam'd him from the nave to the chaps,
And choke their art. The merciless Macdonwald--
Of kerns and gallowglasses is supplied;
Worthy to be a rebel, for to that
Doubtful it stood;
Model saved to ./model-test-2
Done training!
Model saved to ./model-test-3
</code></pre></div>
<p dir="auto">3 - Run <code class="notranslate">$ python training.py test 2.txt</code>. The output should be:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Done training!
Model saved to ./model-test-3"><pre class="notranslate"><code class="notranslate">Done training!
Model saved to ./model-test-3
</code></pre></div>
<p dir="auto">4 - Now, change the <code class="notranslate">num_epochs</code> parameter on line 28 from 1 to 2, and run <code class="notranslate">$ python training.py test 2.txt</code> again. In this case, we have something that resembles:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Letting 'I dare not' wait upon 'I would,'
As thou art in desire? Wouldst thou have that
Such I account thy love. Art thou afeard
Which thou esteem'st the ornament of life,
And wakes it now, to look so green and pale
Model saved to ./model-test-3
And live a coward in thine own esteem,
At what it did so freely? From this time
Like the poor cat i' the adage?
Wherein you dress'd yourself? hath it slept since?
To be the same in thine own act and valour
Model saved to ./model-test-4
Done training!
Model saved to ./model-test-5"><pre class="notranslate"><code class="notranslate">Letting 'I dare not' wait upon 'I would,'
As thou art in desire? Wouldst thou have that
Such I account thy love. Art thou afeard
Which thou esteem'st the ornament of life,
And wakes it now, to look so green and pale
Model saved to ./model-test-3
And live a coward in thine own esteem,
At what it did so freely? From this time
Like the poor cat i' the adage?
Wherein you dress'd yourself? hath it slept since?
To be the same in thine own act and valour
Model saved to ./model-test-4
Done training!
Model saved to ./model-test-5
</code></pre></div>
<p dir="auto">5 - To cleanup, run <code class="notranslate">$ rm checkpoint model-test-*</code>.</p>
<p dir="auto">There are two problems: First, the model does not continue to "train" in step 3. Second, when the <code class="notranslate">num_epochs</code> parameter is modified and it does continue to train, it doesn't reproduce the input twice, as it should (step 4).</p>
<p dir="auto">Playing around with this, I notice that the initial value of <code class="notranslate">num_epochs</code> works as expected, e.g. if at step 1, <code class="notranslate">num_epochs</code> is set to 3, three copies of the input file will be produced. Further, step 4 yields similar results when the new value for <code class="notranslate">num_epochs</code> is larger than the previous value.</p>
<p dir="auto">The <a href="https://github.com/tensorflow/docs/blob/master/site/en/r2/guide/upgrade.md"><code class="notranslate">tf_upgrade_v2</code></a> tool fails to parse commands with common IPython functions (for example, <code class="notranslate">!pip install tf-nightly</code> and <code class="notranslate">%matplotlib inline</code>).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR: Failed to parse.
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.6/dist-packages/tensorflow/tools/compatibility/ast_edits.py&quot;, line 510, in update_string_pasta
    t = pasta.parse(text)
  File &quot;/usr/local/lib/python3.6/dist-packages/pasta/__init__.py&quot;, line 23, in parse
    t = ast_utils.parse(src)
  File &quot;/usr/local/lib/python3.6/dist-packages/pasta/base/ast_utils.py&quot;, line 56, in parse
    tree = ast.parse(sanitize_source(src))
  File &quot;/usr/lib/python3.6/ast.py&quot;, line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File &quot;&lt;unknown&gt;&quot;, line 37
    !pip install tf-nightly
    ^
SyntaxError: invalid syntax"><pre class="notranslate"><code class="notranslate">ERROR: Failed to parse.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/tools/compatibility/ast_edits.py", line 510, in update_string_pasta
    t = pasta.parse(text)
  File "/usr/local/lib/python3.6/dist-packages/pasta/__init__.py", line 23, in parse
    t = ast_utils.parse(src)
  File "/usr/local/lib/python3.6/dist-packages/pasta/base/ast_utils.py", line 56, in parse
    tree = ast.parse(sanitize_source(src))
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "&lt;unknown&gt;", line 37
    !pip install tf-nightly
    ^
SyntaxError: invalid syntax
</code></pre></div>