<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):WINDOWS 7 64</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): source</li>
<li>TensorFlow version: r1.11</li>
<li>Python version: 3.7</li>
<li>Installed using virtualenv? pip? conda?: PIP</li>
<li>Bazel version (if compiling from source): 0.18.0</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version:  not using</li>
<li>GPU model and memory:</li>
</ul>
<p dir="auto">I cant build tensorflow because happens some error about@png_archive</p>
<p dir="auto"><code class="notranslate">C:\Users\00\tensorflow&gt;bazel build --config=opt //tensorflow/tools/pip_package: uild_pip_package WARNING: Processed legacy workspace file c:\users\00\tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https //github.com/bazelbuild/bazel/issues/6319 for further information, including ho to upgrade. WARNING: Option 'experimental_shortened_obj_file_path' is deprecated Loading: Loading: 0 packages loaded INFO: Build options have changed, discarding analysis cache. Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) </code></p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li>
<p dir="auto"><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</p>
</li>
<li>
<p dir="auto"><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Linux Ubuntu 16.04.5 LTS</p>
</li>
<li>
<p dir="auto"><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy)</strong>: N/A</p>
</li>
<li>
<p dir="auto"><strong>TensorFlow installed from (source or binary)</strong>: binary</p>
</li>
<li>
<p dir="auto"><strong>TensorFlow version (use command below)</strong>: v1.10.1-0-g4dcfddc5d1 1.10.1</p>
</li>
<li>
<p dir="auto"><strong>Python version</strong>: 3.6.0</p>
</li>
<li>
<p dir="auto"><strong>Bazel version (if compiling from source)</strong>: N/A</p>
</li>
<li>
<p dir="auto"><strong>CUDA/cuDNN version</strong>:<br>
CUDA Version 9.0.176<br>
cudnn: 7.1.4</p>
</li>
<li>
<p dir="auto"><strong>GPU model and memory</strong>:<br>
NVIDIA GeForce GTX 1080 Ti<br>
11GB</p>
</li>
<li>
<p dir="auto"><strong>Exact command to reproduce</strong>:<br>
<code class="notranslate">python test.py</code></p>
</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto"><strong>Description</strong><br>
If 2 instances of <code class="notranslate">tensorflow.contrib.cudnn_rnn.CudnnLSTM</code> are built via <code class="notranslate">build()</code> method they can not be saved together in a checkpoint.</p>
<p dir="auto"><strong>Bug borders</strong><br>
If instances are built implicitly in <code class="notranslate">__call__()</code>, no exceptions are thrown. Next code executes OK.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import os
import tensorflow as tf
from tensorflow.contrib.cudnn_rnn import CudnnLSTM as CudnnLSTM
inp = tf.zeros([10, 32, 100])
lstm1 = CudnnLSTM(1, 128)
lstm2 = CudnnLSTM(2, 256)
lstm1(inp)
lstm2(inp)
saver = tf.train.Saver()
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    save_path = os.path.join('test_cudnn_lstm_save', '1')
    if not os.path.exists(save_path):
        os.makedirs(os.path.join(save_path))
    saver.save(sess, save_path)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">os</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">from</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">cudnn_rnn</span> <span class="pl-k">import</span> <span class="pl-v">CudnnLSTM</span> <span class="pl-k">as</span> <span class="pl-v">CudnnLSTM</span>
<span class="pl-s1">inp</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">zeros</span>([<span class="pl-c1">10</span>, <span class="pl-c1">32</span>, <span class="pl-c1">100</span>])
<span class="pl-s1">lstm1</span> <span class="pl-c1">=</span> <span class="pl-v">CudnnLSTM</span>(<span class="pl-c1">1</span>, <span class="pl-c1">128</span>)
<span class="pl-s1">lstm2</span> <span class="pl-c1">=</span> <span class="pl-v">CudnnLSTM</span>(<span class="pl-c1">2</span>, <span class="pl-c1">256</span>)
<span class="pl-en">lstm1</span>(<span class="pl-s1">inp</span>)
<span class="pl-en">lstm2</span>(<span class="pl-s1">inp</span>)
<span class="pl-s1">saver</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">Saver</span>()
<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
    <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">tf</span>.<span class="pl-en">global_variables_initializer</span>())
    <span class="pl-s1">save_path</span> <span class="pl-c1">=</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(<span class="pl-s">'test_cudnn_lstm_save'</span>, <span class="pl-s">'1'</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">exists</span>(<span class="pl-s1">save_path</span>):
        <span class="pl-s1">os</span>.<span class="pl-en">makedirs</span>(<span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(<span class="pl-s1">save_path</span>))
    <span class="pl-s1">saver</span>.<span class="pl-en">save</span>(<span class="pl-s1">sess</span>, <span class="pl-s1">save_path</span>)</pre></div>
<p dir="auto">Also if only 1 instance is built using <code class="notranslate">build()</code> or if only 1 instance is passed to saver in <code class="notranslate">var_list</code> (see exсerpt below), save op is executed without exceptions.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="saver = tf.train.Saver([lstm1.saveable])"><pre class="notranslate"><span class="pl-s1">saver</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">Saver</span>([<span class="pl-s1">lstm1</span>.<span class="pl-s1">saveable</span>])</pre></div>
<p dir="auto"><strong>Relevance</strong><br>
I ran into this problem when tried to train my net on several GPUs. Shared CudnnLSTM params are built on CPU, while computations are made on GPUs.</p>
<p dir="auto">Thank you in advance for any help!</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">Code to reproduce the bug</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import os
import tensorflow as tf
from tensorflow.contrib.cudnn_rnn import CudnnLSTM as CudnnLSTM
inp = tf.zeros([10, 32, 100])
lstm1 = CudnnLSTM(1, 128)
lstm2 = CudnnLSTM(2, 256)
lstm1.build(inp.shape)
lstm2.build(inp.shape)
saver = tf.train.Saver()
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    save_path = os.path.join('test_cudnn_lstm_save', '1')
    if not os.path.exists(save_path):
        os.makedirs(os.path.join(save_path))
    saver.save(sess, save_path)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">os</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">from</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">cudnn_rnn</span> <span class="pl-k">import</span> <span class="pl-v">CudnnLSTM</span> <span class="pl-k">as</span> <span class="pl-v">CudnnLSTM</span>
<span class="pl-s1">inp</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">zeros</span>([<span class="pl-c1">10</span>, <span class="pl-c1">32</span>, <span class="pl-c1">100</span>])
<span class="pl-s1">lstm1</span> <span class="pl-c1">=</span> <span class="pl-v">CudnnLSTM</span>(<span class="pl-c1">1</span>, <span class="pl-c1">128</span>)
<span class="pl-s1">lstm2</span> <span class="pl-c1">=</span> <span class="pl-v">CudnnLSTM</span>(<span class="pl-c1">2</span>, <span class="pl-c1">256</span>)
<span class="pl-s1">lstm1</span>.<span class="pl-en">build</span>(<span class="pl-s1">inp</span>.<span class="pl-s1">shape</span>)
<span class="pl-s1">lstm2</span>.<span class="pl-en">build</span>(<span class="pl-s1">inp</span>.<span class="pl-s1">shape</span>)
<span class="pl-s1">saver</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">Saver</span>()
<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
    <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">tf</span>.<span class="pl-en">global_variables_initializer</span>())
    <span class="pl-s1">save_path</span> <span class="pl-c1">=</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(<span class="pl-s">'test_cudnn_lstm_save'</span>, <span class="pl-s">'1'</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">exists</span>(<span class="pl-s1">save_path</span>):
        <span class="pl-s1">os</span>.<span class="pl-en">makedirs</span>(<span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(<span class="pl-s1">save_path</span>))
    <span class="pl-s1">saver</span>.<span class="pl-en">save</span>(<span class="pl-s1">sess</span>, <span class="pl-s1">save_path</span>)</pre></div>
<p dir="auto">Here is a traceback</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-10-04 12:59:46.556727: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-04 12:59:46.635329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 12:59:46.635669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 10.92GiB freeMemory: 10.01GiB
2018-10-04 12:59:46.635682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-10-04 12:59:46.841898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 12:59:46.841930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2018-10-04 12:59:46.841937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2018-10-04 12:59:46.842099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9673 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 12:59:47.465382: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at save_restore_v2_ops.cc:134 : Invalid argument: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
Traceback (most recent call last):
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1278, in _do_call
    return fn(*args)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1263, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1350, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;test.py&quot;, line 17, in &lt;module&gt;
    saver.save(sess, save_path)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 1620, in save
    {self.saver_def.filename_tensor_name: checkpoint_file})
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 877, in run
    run_metadata_ptr)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1272, in _do_run
    run_metadata)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py&quot;, line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

Caused by op 'save/SaveV2', defined at:
  File &quot;test.py&quot;, line 11, in &lt;module&gt;
    saver = tf.train.Saver()
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 1281, in __init__
    self.build()
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 1293, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 1330, in _build
    build_save=build_save, build_restore=build_restore)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 775, in _build_internal
    save_tensor = self._AddSaveOps(filename_tensor, saveables)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 275, in _AddSaveOps
    save = self.save_op(filename_tensor, saveables)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py&quot;, line 193, in save_op
    tensors)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py&quot;, line 1687, in save_v2
    shape_and_slices=shape_and_slices, tensors=tensors, name=name)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py&quot;, line 787, in _apply_op_helper
    op_def=op_def)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py&quot;, line 454, in new_func
    return func(*args, **kwargs)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py&quot;, line 3155, in create_op
    op_def=op_def)
  File &quot;/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py&quot;, line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]"><pre class="notranslate"><code class="notranslate">2018-10-04 12:59:46.556727: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-04 12:59:46.635329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 12:59:46.635669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 10.92GiB freeMemory: 10.01GiB
2018-10-04 12:59:46.635682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-10-04 12:59:46.841898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 12:59:46.841930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2018-10-04 12:59:46.841937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2018-10-04 12:59:46.842099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9673 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 12:59:47.465382: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at save_restore_v2_ops.cc:134 : Invalid argument: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
Traceback (most recent call last):
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
    return fn(*args)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 17, in &lt;module&gt;
    saver.save(sess, save_path)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1620, in save
    {self.saver_def.filename_tensor_name: checkpoint_file})
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

Caused by op 'save/SaveV2', defined at:
  File "test.py", line 11, in &lt;module&gt;
    saver = tf.train.Saver()
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1281, in __init__
    self.build()
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1293, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1330, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 775, in _build_internal
    save_tensor = self._AddSaveOps(filename_tensor, saveables)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 275, in _AddSaveOps
    save = self.save_op(filename_tensor, saveables)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 193, in save_op
    tensors)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1687, in save_v2
    shape_and_slices=shape_and_slices, tensors=tensors, name=name)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]
</code></pre></div>