<p dir="auto">This is a feature request.</p>
<p dir="auto">Let's consider a scenario where one trains multiple models and uses them in combination (like it is done in GANs).</p>
<p dir="auto">To simplify the process of saving and restoring variables that are partly shared across the models (Pretraining Model, Training Model, Evaluation Model, Infer Model) one could instantiate the whole graph, containing all operations and variables, and save it.</p>
<p dir="auto">Then in order to do pre-training only the subset of graph elements that is required for pretraining is used.</p>
<p dir="auto">This results in the overhead of having to build the whole model (which might consist of multiple sub models) the first time the model is run even though only a smaller part (e.g. Pretraining Model) is required.</p>
<p dir="auto">Another issue arises when using different optimizers in different training stages (e.g. SGD first then Adam). As Adam creates additional variables Adam has to be instantiated during the first training stage so restoring from a checkpoint does not fail when restoring with Adam instead of SGD.</p>
<p dir="auto">This restriction of having to build everything despite parts not being required results in more complicated code. If it would be possible to silently fail when a variable is not found in a checkpoint, so it can be initialized with <code class="notranslate">tf.global_variables_initializer()</code> instead, would allow better structuring of code.</p>
<p dir="auto">I have looked through all current issues regarding this problem and I have found a couple that face a similar problem and where a <code class="notranslate">QuietlyFailRestoringSaver</code> could solve this problem:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="247946712" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/12032" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/12032/hovercard" href="https://github.com/tensorflow/tensorflow/issues/12032">#12032</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="294537159" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/16781" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/16781/hovercard" href="https://github.com/tensorflow/tensorflow/issues/16781">#16781</a></p>
<p dir="auto">I might consider building this if there is enough support for it. I am open for feedback.</p>
<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA</li>
<li>TensorFlow installed from (source or binary): 1.13.1 from conda</li>
<li>TensorFlow version (use command below): 1.13.1</li>
<li>Python version: 3.7.3</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version:</li>
<li>GPU model and memory:</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture <a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with<br>
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf

def foo(x):
    array = tf.Variable(lambda: tf.zeros(20, dtype=tf.float32),
            trainable=False, use_resource=True)
    ind = tf.reshape(x, [-1, 1])
    val = tf.reshape(tf.cast(x, tf.float32), [-1])

    array_assign = array.assign(tf.zeros_like(array))
    with tf.control_dependencies([array_assign]):
        array = array.scatter_nd_update(ind, tf.cast(val, dtype=tf.float32))

    return array

dataset = tf.data.Dataset.range(20)
# dataset = dataset.map(foo)
# dataset = dataset.batch(5)
dataset = dataset.apply(
        tf.data.experimental.map_and_batch(foo, 5, num_parallel_batches=1))

iterator = dataset.make_initializable_iterator()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(iterator.initializer)
    next_element = iterator.get_next()
    fetches = sess.run(next_element)
    print(fetches)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf

def foo(x):
    array = tf.Variable(lambda: tf.zeros(20, dtype=tf.float32),
            trainable=False, use_resource=True)
    ind = tf.reshape(x, [-1, 1])
    val = tf.reshape(tf.cast(x, tf.float32), [-1])

    array_assign = array.assign(tf.zeros_like(array))
    with tf.control_dependencies([array_assign]):
        array = array.scatter_nd_update(ind, tf.cast(val, dtype=tf.float32))

    return array

dataset = tf.data.Dataset.range(20)
# dataset = dataset.map(foo)
# dataset = dataset.batch(5)
dataset = dataset.apply(
        tf.data.experimental.map_and_batch(foo, 5, num_parallel_batches=1))

iterator = dataset.make_initializable_iterator()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(iterator.initializer)
    next_element = iterator.get_next()
    fetches = sess.run(next_element)
    print(fetches)
</code></pre></div>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">If using normal <code class="notranslate">map</code> and <code class="notranslate">batch</code>, one get</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 3. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]"><pre class="notranslate"><code class="notranslate">[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 3. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
</code></pre></div>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">If using <code class="notranslate">map_and_batch</code>, one get changing results like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[[0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]"><pre class="notranslate"><code class="notranslate">[[0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
</code></pre></div>
<p dir="auto">There should be always only one and non-duplicate non-zeros per-line.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto">Not sure if this is related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="429263088" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/27507" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/27507/hovercard" href="https://github.com/tensorflow/tensorflow/issues/27507">#27507</a> . On a <a href="https://colab.research.google.com/drive/1DzTvOJRXYUk0CSjvGa11ewmjTP7VJ1kF" rel="nofollow">Colab with tf-nightly</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="429263088" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/27507" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/27507/hovercard" href="https://github.com/tensorflow/tensorflow/issues/27507">#27507</a> seems fixed but this issue still reproduces.</p>