<p dir="auto">I am facing similar issue but for GRU. I am using tensorflow  1.1.0 and I tried dumping the model in different ways:<br>
a)          saver = tf.train.Saver(tf.global_variables())<br>
model_exporter = exporter.Exporter(saver)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    # Restore variables from training checkpoint
    # TODO: This restores the most recent checkpoint, but if we use validation to counterract
    #       over-fitting, we may want to restore an earlier checkpoint.
    checkpoint = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)
    checkpoint_path = checkpoint.model_checkpoint_path
    saver.restore(session, checkpoint_path)
    log_info('Restored checkpoint at training epoch %d' % (int(checkpoint_path.split('-')[-1]) + 1))

    # Initialise the model exporter and export the model
    model_exporter.init(session.graph.as_graph_def(),
                        named_graph_signatures = {
                            'inputs': exporter.generic_signature(
                                { 'input': input_tensor,
                                  'input_lengths': seq_length}),
                            'outputs': exporter.generic_signature(
                                { 'outputs': decoded})})
    if FLAGS.remove_export:
        actual_export_dir = os.path.join(FLAGS.export_dir, '%08d' % FLAGS.export_version)
        if os.path.isdir(actual_export_dir):
            log_info('Removing old export')
            shutil.rmtree(actual_FLAGS.export_dir)
    try:
        # Export serving model
        model_exporter.export(FLAGS.export_dir, tf.constant(FLAGS.export_version), session)

        # Export graph
        input_graph_name = 'input_graph.pb'
        tf.train.write_graph(session.graph, FLAGS.export_dir, input_graph_name, as_text=False)

        # Freeze graph
        input_graph_path = os.path.join(FLAGS.export_dir, input_graph_name)
        input_saver_def_path = ''
        input_binary = True
        output_node_names = 'output_node'
        restore_op_name = 'save/restore_all'
        filename_tensor_name = 'save/Const:0'
        output_graph_path = os.path.join(FLAGS.export_dir, 'output_graph.pb')
        clear_devices = False
        freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
                                  input_binary, checkpoint_path, output_node_names,
                                  restore_op_name, filename_tensor_name,
                                  output_graph_path, clear_devices, '')"><pre class="notranslate"><code class="notranslate">    # Restore variables from training checkpoint
    # TODO: This restores the most recent checkpoint, but if we use validation to counterract
    #       over-fitting, we may want to restore an earlier checkpoint.
    checkpoint = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)
    checkpoint_path = checkpoint.model_checkpoint_path
    saver.restore(session, checkpoint_path)
    log_info('Restored checkpoint at training epoch %d' % (int(checkpoint_path.split('-')[-1]) + 1))

    # Initialise the model exporter and export the model
    model_exporter.init(session.graph.as_graph_def(),
                        named_graph_signatures = {
                            'inputs': exporter.generic_signature(
                                { 'input': input_tensor,
                                  'input_lengths': seq_length}),
                            'outputs': exporter.generic_signature(
                                { 'outputs': decoded})})
    if FLAGS.remove_export:
        actual_export_dir = os.path.join(FLAGS.export_dir, '%08d' % FLAGS.export_version)
        if os.path.isdir(actual_export_dir):
            log_info('Removing old export')
            shutil.rmtree(actual_FLAGS.export_dir)
    try:
        # Export serving model
        model_exporter.export(FLAGS.export_dir, tf.constant(FLAGS.export_version), session)

        # Export graph
        input_graph_name = 'input_graph.pb'
        tf.train.write_graph(session.graph, FLAGS.export_dir, input_graph_name, as_text=False)

        # Freeze graph
        input_graph_path = os.path.join(FLAGS.export_dir, input_graph_name)
        input_saver_def_path = ''
        input_binary = True
        output_node_names = 'output_node'
        restore_op_name = 'save/restore_all'
        filename_tensor_name = 'save/Const:0'
        output_graph_path = os.path.join(FLAGS.export_dir, 'output_graph.pb')
        clear_devices = False
        freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
                                  input_binary, checkpoint_path, output_node_names,
                                  restore_op_name, filename_tensor_name,
                                  output_graph_path, clear_devices, '')
</code></pre></div>
<p dir="auto">b)          output_graph_def = graph_util.convert_variables_to_constants(session, session.graph.as_graph_def(), ['output_node'])<br>
with gfile.FastGFile('./data/ldc93s1/output_graph2.pb', 'wb') as f:<br>
f.write(output_graph_def.SerializeToString())</p>
<p dir="auto">but for both the dump I get the following error: -<br>
'rnn/while/multi_rnn_cell/cell_0/gru_cell/gates/r/cond/rnn/while/multi_rnn_cell/cell_0/gru_cell/gates/r/strided_slice/_assign/RefEnter': Input tensor 'rnn/multi_rnn_cell/cell_0/gru_cell/gates/r/pop_mean:0' <strong>Cannot convert a tensor of type float32 to an input of type float32_ref</strong></p>
<p dir="auto">Any solution so far?<br>
I followed the similar bug but that is specifically related to Batch norm <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="169195660" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/3628" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/3628/hovercard" href="https://github.com/tensorflow/tensorflow/issues/3628">#3628</a> . And what is the reason behind this??</p>
<p dir="auto">Please go to Stack Overflow for help and support:</p>
<p dir="auto"><a href="https://stackoverflow.com/questions/tagged/tensorflow" rel="nofollow">https://stackoverflow.com/questions/tagged/tensorflow</a></p>
<p dir="auto">If you open a GitHub issue, here is our policy:</p>
<ol dir="auto">
<li>It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).</li>
<li>The form below must be filled out.</li>
<li>It shouldn't be a TensorBoard issue. Those go <a href="https://github.com/tensorflow/tensorboard/issues">here</a>.</li>
</ol>
<p dir="auto"><strong>Here's why we have that policy</strong>: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.</p>
<hr>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:</li>
<li><strong>TensorFlow version (use command below)</strong>:</li>
<li><strong>Python version</strong>:</li>
<li><strong>Bazel version (if compiling from source)</strong>:</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:</li>
<li><strong>CUDA/cuDNN version</strong>:</li>
<li><strong>GPU model and memory</strong>:</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture script:</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh</a></p>
<p dir="auto">You can obtain the TensorFlow version with</p>
<p dir="auto">python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Reporting issues raise in this stackoverflow question:<br>
<a href="https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator" rel="nofollow">https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator</a>:</p>
<ol dir="auto">
<li>tf.train.init_from_checkpoint doesn't work well with MirroredStrategy</li>
<li>tf.estimator.WarmStartSettings with Estimator doesn't work with MirroredStrategy</li>
</ol>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">See code snippets and error messages in this stack overflow page:<br>
<a href="https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator" rel="nofollow">https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator</a></p>
<p dir="auto">The root cause is that we haven't modified these 2 paths for restoring from checkpoints to work with mirrored variables. So we need to do 2 things:</p>
<ol dir="auto">
<li>Use the appropriate method to restore mirrored variables in these codepaths - either via distribution.update or by using Saver (see Allen's suggestion below).</li>
<li>if the restoring is happening in tower context(for e.g. when init_from_checkpoint is called in model_fn), then we may need to first enter cross tower context before restoring.</li>
</ol>
<p dir="auto">Suggestion for addressing <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="115886302" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/1" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/1/hovercard" href="https://github.com/tensorflow/tensorflow/issues/1">#1</a> from <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/allenlavoie/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/allenlavoie">@allenlavoie</a> :<br>
Do the same <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/saver.py#L531">SaveableObject extraction as Saver</a> in warm_start, then just call SaveableObject.restore() with the Tensor instead of using .assign. This will work for MirroredVariables and whatever else is checkpointable (and means that to work with both, new objects only need to do the checkpointable thing). Potentially we could just call <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/saver.py#L573">SaveableObjectsForOp</a> for everything including variables.</p>