<p dir="auto">I am installing tensorflow with GPU and meeting some problems:</p>
<p dir="auto">WARNING: /home/EI/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead.<br>
WARNING:  #/home/EI/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead.<br>
INFO: Found 1 target...<br>
ERROR: /home/EI/tensorflow/tensorflow/core/BUILD:1280:1: C++ compilation of rule '//tensorflow/core:lib_hash_crc32c_accelerate_internal' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 41 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.<br>
gcc: error trying to exec 'cc1plus': execvp: No such file or directory<br>
Target //tensorflow/tools/pip_package:build_pip_package failed to build<br>
Use --verbose_failures to see the command lines of failed build steps.<br>
INFO: Elapsed time: 1994.312s, Critical Path: 8.68s</p>
<p dir="auto">Operating System: Ubuntu 16.04LTS<br>
gcc  4.9.3<br>
bazel :0.4.5<br>
python:3.6.0<br>
CUDA:8.0.61_375.62<br>
cuDNN:5.1.1</p>
<p dir="auto">Currently it is not possible to scope all variables cleanly when using RNN cells and Optimizers. RNN cells create variables with tf.get_variable and optimizers create variables with tf.Variable. These two methods don't get along, creating messy variable names when variable scopes are handled differently by the two methods.</p>
<p dir="auto">Script for testing different variable creation and scoping combinations.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# -*- coding: utf-8 -*-

import tensorflow as tf


def build_loss(use_get_variable=True):
    y = tf.placeholder('int32', [None], name='y')
    x = tf.placeholder('float32', shape=[None, None, 10], name='x')
    cell = tf.contrib.rnn.GRUCell(128)  # RNN cell
    if use_get_variable:
        # Create variable with tf.get_variable
        w = tf.get_variable('w', dtype='float32', initializer=tf.random_normal([128, 10]))
    else:
        # Create variable with tf.Variable
        w = tf.Variable(initial_value=tf.random_normal([128, 10]), dtype='float32', name='w')
    rnn_out, _ = tf.nn.dynamic_rnn(cell, x, dtype='float32', time_major=False)  # RNN
    rnn_out = rnn_out[:, -1, :]  # Last timestep
    rnn_out = tf.matmul(rnn_out, w)  # Output layer projection
    loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=rnn_out))  # Cross entropy
    return loss_op


def inspect(use_get_variable=True, scope_optimizer=True, re_enter_scope=None, optimizer=tf.train.AdamOptimizer):
    scope = 'scope'
    if re_enter_scope:
        with tf.variable_scope(scope) as variable_scope:  # Capture variable scope
            if re_enter_scope == 'object':
                # Save captured object for re-entering scope with captured variable scope object
                scope = variable_scope
            elif re_enter_scope == 'original_name_scope':
                # Save original name scope of captured variable scope for re-entering the scope with original name scope
                scope = variable_scope.original_name_scope
    with tf.variable_scope(scope):  # (Re-)enter scope
        loss_op = build_loss(use_get_variable=use_get_variable)
        if scope_optimizer:
            # Create optimizer in the variable scope
            train_op = optimizer(0.1).minimize(loss_op)
    if not scope_optimizer:
        # Create optimizer outside of variable scope
        train_op = optimizer(0.1).minimize(loss_op)
    with tf.Session() as sess:
        # Initialize variables
        sess.run(tf.global_variables_initializer())

    description = 'optimizer ' + ('NOT' if not scope_optimizer else 'IS') + ' scoped, '
    if not re_enter_scope:
        description += 'using ORIGINAL scope, '
    elif re_enter_scope == 'object':
        description += 're-entering scope with OBJECT, '
    elif re_enter_scope == 'original_name_scope':
        description += 're-entering scope with ORIGINAL_NAME_SCOPE, '
    description += 'variables created with ' + ('tf.get_variable' if use_get_variable else 'tf.Variable')
    print(description)

    print('\n'.join(['  '+var.name for var in tf.global_variables()]))
    print()

    tf.reset_default_graph()


def main():
    # Ideally scope everything with re-entered scope, leads to double scoping of gradient variables and having duplicate
    # scope (with unique name) for Adam optimizer beta power variables
    inspect(use_get_variable=True, scope_optimizer=True, re_enter_scope='object')
    # Not scoping optimizer fixes double scoping of RNN variables but leads to having AdamOptimizer beta_powers not
    # scoped
    inspect(use_get_variable=True, scope_optimizer=False, re_enter_scope='object')
    # Using tf.Variable for variable creation when re-entering scope with scope object leads to creation of duplicate
    # scope (with unique name) for non-RNN variables
    inspect(use_get_variable=False, scope_optimizer=False, re_enter_scope='object')
    # Changing scope re-entering method to variable_scope.original_name_scope fixes the problem of duplicate scope
    # when using tf.Variable for variable creation, but creates additional problem of having double slashes in
    # scope hierarchy path with RNN variables.
    inspect(use_get_variable=False, scope_optimizer=False, re_enter_scope='original_name_scope')
    # Similar problem is observed when using tf.get_variable for variable creation and re-entering scope with
    # original_name_scope but this time double slashes are observed also with non-RNN variables.
    inspect(use_get_variable=True, scope_optimizer=False, re_enter_scope='original_name_scope')
    # Scoping optimizer without re-entering the scope fixes scoping of Adam optimizer beta power variables but does not
    # fix the problem of having double scopes for RNN variables. However re-entering scopes is desired for OOP.
    inspect(use_get_variable=False, scope_optimizer=True, re_enter_scope=False)
    # Using tf.get_variable for variable creation won't help since Optimizer creates variables with tf.Variable and RNN
    # cells create variables with tf.get_variable, these two methods don't get along
    inspect(use_get_variable=True, scope_optimizer=True, re_enter_scope=False)
    # Swapping AdamOptimizer for RMSPropOptimizer fixes the problem with AdamOptimizer's beta power variables
    inspect(use_get_variable=True, scope_optimizer=False, re_enter_scope='object', optimizer=tf.train.RMSPropOptimizer)


if __name__ == '__main__':
    main()"><pre class="notranslate"><span class="pl-c"># -*- coding: utf-8 -*-</span>

<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>


<span class="pl-k">def</span> <span class="pl-en">build_loss</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s">'int32'</span>, [<span class="pl-c1">None</span>], <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'y'</span>)
    <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s">'float32'</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">None</span>, <span class="pl-c1">None</span>, <span class="pl-c1">10</span>], <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'x'</span>)
    <span class="pl-s1">cell</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">rnn</span>.<span class="pl-v">GRUCell</span>(<span class="pl-c1">128</span>)  <span class="pl-c"># RNN cell</span>
    <span class="pl-k">if</span> <span class="pl-s1">use_get_variable</span>:
        <span class="pl-c"># Create variable with tf.get_variable</span>
        <span class="pl-s1">w</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_variable</span>(<span class="pl-s">'w'</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'float32'</span>, <span class="pl-s1">initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">random_normal</span>([<span class="pl-c1">128</span>, <span class="pl-c1">10</span>]))
    <span class="pl-k">else</span>:
        <span class="pl-c"># Create variable with tf.Variable</span>
        <span class="pl-s1">w</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-s1">initial_value</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">random_normal</span>([<span class="pl-c1">128</span>, <span class="pl-c1">10</span>]), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'float32'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'w'</span>)
    <span class="pl-s1">rnn_out</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">dynamic_rnn</span>(<span class="pl-s1">cell</span>, <span class="pl-s1">x</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'float32'</span>, <span class="pl-s1">time_major</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)  <span class="pl-c"># RNN</span>
    <span class="pl-s1">rnn_out</span> <span class="pl-c1">=</span> <span class="pl-s1">rnn_out</span>[:, <span class="pl-c1">-</span><span class="pl-c1">1</span>, :]  <span class="pl-c"># Last timestep</span>
    <span class="pl-s1">rnn_out</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">matmul</span>(<span class="pl-s1">rnn_out</span>, <span class="pl-s1">w</span>)  <span class="pl-c"># Output layer projection</span>
    <span class="pl-s1">loss_op</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">sparse_softmax_cross_entropy_with_logits</span>(<span class="pl-s1">labels</span><span class="pl-c1">=</span><span class="pl-s1">y</span>, <span class="pl-s1">logits</span><span class="pl-c1">=</span><span class="pl-s1">rnn_out</span>))  <span class="pl-c"># Cross entropy</span>
    <span class="pl-k">return</span> <span class="pl-s1">loss_op</span>


<span class="pl-k">def</span> <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">optimizer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">AdamOptimizer</span>):
    <span class="pl-s1">scope</span> <span class="pl-c1">=</span> <span class="pl-s">'scope'</span>
    <span class="pl-k">if</span> <span class="pl-s1">re_enter_scope</span>:
        <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s1">scope</span>) <span class="pl-k">as</span> <span class="pl-s1">variable_scope</span>:  <span class="pl-c"># Capture variable scope</span>
            <span class="pl-k">if</span> <span class="pl-s1">re_enter_scope</span> <span class="pl-c1">==</span> <span class="pl-s">'object'</span>:
                <span class="pl-c"># Save captured object for re-entering scope with captured variable scope object</span>
                <span class="pl-s1">scope</span> <span class="pl-c1">=</span> <span class="pl-s1">variable_scope</span>
            <span class="pl-k">elif</span> <span class="pl-s1">re_enter_scope</span> <span class="pl-c1">==</span> <span class="pl-s">'original_name_scope'</span>:
                <span class="pl-c"># Save original name scope of captured variable scope for re-entering the scope with original name scope</span>
                <span class="pl-s1">scope</span> <span class="pl-c1">=</span> <span class="pl-s1">variable_scope</span>.<span class="pl-s1">original_name_scope</span>
    <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s1">scope</span>):  <span class="pl-c"># (Re-)enter scope</span>
        <span class="pl-s1">loss_op</span> <span class="pl-c1">=</span> <span class="pl-en">build_loss</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-s1">use_get_variable</span>)
        <span class="pl-k">if</span> <span class="pl-s1">scope_optimizer</span>:
            <span class="pl-c"># Create optimizer in the variable scope</span>
            <span class="pl-s1">train_op</span> <span class="pl-c1">=</span> <span class="pl-en">optimizer</span>(<span class="pl-c1">0.1</span>).<span class="pl-en">minimize</span>(<span class="pl-s1">loss_op</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">scope_optimizer</span>:
        <span class="pl-c"># Create optimizer outside of variable scope</span>
        <span class="pl-s1">train_op</span> <span class="pl-c1">=</span> <span class="pl-en">optimizer</span>(<span class="pl-c1">0.1</span>).<span class="pl-en">minimize</span>(<span class="pl-s1">loss_op</span>)
    <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
        <span class="pl-c"># Initialize variables</span>
        <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">tf</span>.<span class="pl-en">global_variables_initializer</span>())

    <span class="pl-s1">description</span> <span class="pl-c1">=</span> <span class="pl-s">'optimizer '</span> <span class="pl-c1">+</span> (<span class="pl-s">'NOT'</span> <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">scope_optimizer</span> <span class="pl-k">else</span> <span class="pl-s">'IS'</span>) <span class="pl-c1">+</span> <span class="pl-s">' scoped, '</span>
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">re_enter_scope</span>:
        <span class="pl-s1">description</span> <span class="pl-c1">+=</span> <span class="pl-s">'using ORIGINAL scope, '</span>
    <span class="pl-k">elif</span> <span class="pl-s1">re_enter_scope</span> <span class="pl-c1">==</span> <span class="pl-s">'object'</span>:
        <span class="pl-s1">description</span> <span class="pl-c1">+=</span> <span class="pl-s">'re-entering scope with OBJECT, '</span>
    <span class="pl-k">elif</span> <span class="pl-s1">re_enter_scope</span> <span class="pl-c1">==</span> <span class="pl-s">'original_name_scope'</span>:
        <span class="pl-s1">description</span> <span class="pl-c1">+=</span> <span class="pl-s">'re-entering scope with ORIGINAL_NAME_SCOPE, '</span>
    <span class="pl-s1">description</span> <span class="pl-c1">+=</span> <span class="pl-s">'variables created with '</span> <span class="pl-c1">+</span> (<span class="pl-s">'tf.get_variable'</span> <span class="pl-k">if</span> <span class="pl-s1">use_get_variable</span> <span class="pl-k">else</span> <span class="pl-s">'tf.Variable'</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">description</span>)

    <span class="pl-en">print</span>(<span class="pl-s">'<span class="pl-cce">\n</span>'</span>.<span class="pl-en">join</span>([<span class="pl-s">'  '</span><span class="pl-c1">+</span><span class="pl-s1">var</span>.<span class="pl-s1">name</span> <span class="pl-k">for</span> <span class="pl-s1">var</span> <span class="pl-c1">in</span> <span class="pl-s1">tf</span>.<span class="pl-en">global_variables</span>()]))
    <span class="pl-en">print</span>()

    <span class="pl-s1">tf</span>.<span class="pl-en">reset_default_graph</span>()


<span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-c"># Ideally scope everything with re-entered scope, leads to double scoping of gradient variables and having duplicate</span>
    <span class="pl-c"># scope (with unique name) for Adam optimizer beta power variables</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'object'</span>)
    <span class="pl-c"># Not scoping optimizer fixes double scoping of RNN variables but leads to having AdamOptimizer beta_powers not</span>
    <span class="pl-c"># scoped</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'object'</span>)
    <span class="pl-c"># Using tf.Variable for variable creation when re-entering scope with scope object leads to creation of duplicate</span>
    <span class="pl-c"># scope (with unique name) for non-RNN variables</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'object'</span>)
    <span class="pl-c"># Changing scope re-entering method to variable_scope.original_name_scope fixes the problem of duplicate scope</span>
    <span class="pl-c"># when using tf.Variable for variable creation, but creates additional problem of having double slashes in</span>
    <span class="pl-c"># scope hierarchy path with RNN variables.</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'original_name_scope'</span>)
    <span class="pl-c"># Similar problem is observed when using tf.get_variable for variable creation and re-entering scope with</span>
    <span class="pl-c"># original_name_scope but this time double slashes are observed also with non-RNN variables.</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'original_name_scope'</span>)
    <span class="pl-c"># Scoping optimizer without re-entering the scope fixes scoping of Adam optimizer beta power variables but does not</span>
    <span class="pl-c"># fix the problem of having double scopes for RNN variables. However re-entering scopes is desired for OOP.</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
    <span class="pl-c"># Using tf.get_variable for variable creation won't help since Optimizer creates variables with tf.Variable and RNN</span>
    <span class="pl-c"># cells create variables with tf.get_variable, these two methods don't get along</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
    <span class="pl-c"># Swapping AdamOptimizer for RMSPropOptimizer fixes the problem with AdamOptimizer's beta power variables</span>
    <span class="pl-en">inspect</span>(<span class="pl-s1">use_get_variable</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">scope_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">re_enter_scope</span><span class="pl-c1">=</span><span class="pl-s">'object'</span>, <span class="pl-s1">optimizer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">RMSPropOptimizer</span>)


<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-en">main</span>()</pre></div>
<p dir="auto">outputs</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="optimizer IS scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope_1/beta1_power:0
  scope_1/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope/w/Adam:0
  scope/w/Adam_1:0
  scope/rnn/gru_cell/gates/weights/Adam:0
  scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/rnn/gru_cell/gates/biases/Adam:0
  scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.Variable
  scope_1/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope_1/w/Adam:0
  scope_1/w/Adam_1:0
  scope/rnn/gru_cell/gates/weights/Adam:0
  scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/rnn/gru_cell/gates/biases/Adam:0
  scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with ORIGINAL_NAME_SCOPE, variables created with tf.Variable
  scope/w:0
  scope//rnn/gru_cell/gates/weights:0
  scope//rnn/gru_cell/gates/biases:0
  scope//rnn/gru_cell/candidate/weights:0
  scope//rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope/w/Adam:0
  scope/w/Adam_1:0
  scope//rnn/gru_cell/gates/weights/Adam:0
  scope//rnn/gru_cell/gates/weights/Adam_1:0
  scope//rnn/gru_cell/gates/biases/Adam:0
  scope//rnn/gru_cell/gates/biases/Adam_1:0
  scope//rnn/gru_cell/candidate/weights/Adam:0
  scope//rnn/gru_cell/candidate/weights/Adam_1:0
  scope//rnn/gru_cell/candidate/biases/Adam:0
  scope//rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with ORIGINAL_NAME_SCOPE, variables created with tf.get_variable
  scope//w:0
  scope//rnn/gru_cell/gates/weights:0
  scope//rnn/gru_cell/gates/biases:0
  scope//rnn/gru_cell/candidate/weights:0
  scope//rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope//w/Adam:0
  scope//w/Adam_1:0
  scope//rnn/gru_cell/gates/weights/Adam:0
  scope//rnn/gru_cell/gates/weights/Adam_1:0
  scope//rnn/gru_cell/gates/biases/Adam:0
  scope//rnn/gru_cell/gates/biases/Adam_1:0
  scope//rnn/gru_cell/candidate/weights/Adam:0
  scope//rnn/gru_cell/candidate/weights/Adam_1:0
  scope//rnn/gru_cell/candidate/biases/Adam:0
  scope//rnn/gru_cell/candidate/biases/Adam_1:0

optimizer IS scoped, using ORIGINAL scope, variables created with tf.Variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/beta1_power:0
  scope/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer IS scoped, using ORIGINAL scope, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/beta1_power:0
  scope/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/w/RMSProp:0
  scope/w/RMSProp_1:0
  scope/rnn/gru_cell/gates/weights/RMSProp:0
  scope/rnn/gru_cell/gates/weights/RMSProp_1:0
  scope/rnn/gru_cell/gates/biases/RMSProp:0
  scope/rnn/gru_cell/gates/biases/RMSProp_1:0
  scope/rnn/gru_cell/candidate/weights/RMSProp:0
  scope/rnn/gru_cell/candidate/weights/RMSProp_1:0
  scope/rnn/gru_cell/candidate/biases/RMSProp:0
  scope/rnn/gru_cell/candidate/biases/RMSProp_1:0"><pre class="notranslate"><code class="notranslate">optimizer IS scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope_1/beta1_power:0
  scope_1/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope/w/Adam:0
  scope/w/Adam_1:0
  scope/rnn/gru_cell/gates/weights/Adam:0
  scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/rnn/gru_cell/gates/biases/Adam:0
  scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.Variable
  scope_1/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope_1/w/Adam:0
  scope_1/w/Adam_1:0
  scope/rnn/gru_cell/gates/weights/Adam:0
  scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/rnn/gru_cell/gates/biases/Adam:0
  scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with ORIGINAL_NAME_SCOPE, variables created with tf.Variable
  scope/w:0
  scope//rnn/gru_cell/gates/weights:0
  scope//rnn/gru_cell/gates/biases:0
  scope//rnn/gru_cell/candidate/weights:0
  scope//rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope/w/Adam:0
  scope/w/Adam_1:0
  scope//rnn/gru_cell/gates/weights/Adam:0
  scope//rnn/gru_cell/gates/weights/Adam_1:0
  scope//rnn/gru_cell/gates/biases/Adam:0
  scope//rnn/gru_cell/gates/biases/Adam_1:0
  scope//rnn/gru_cell/candidate/weights/Adam:0
  scope//rnn/gru_cell/candidate/weights/Adam_1:0
  scope//rnn/gru_cell/candidate/biases/Adam:0
  scope//rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with ORIGINAL_NAME_SCOPE, variables created with tf.get_variable
  scope//w:0
  scope//rnn/gru_cell/gates/weights:0
  scope//rnn/gru_cell/gates/biases:0
  scope//rnn/gru_cell/candidate/weights:0
  scope//rnn/gru_cell/candidate/biases:0
  beta1_power:0
  beta2_power:0
  scope//w/Adam:0
  scope//w/Adam_1:0
  scope//rnn/gru_cell/gates/weights/Adam:0
  scope//rnn/gru_cell/gates/weights/Adam_1:0
  scope//rnn/gru_cell/gates/biases/Adam:0
  scope//rnn/gru_cell/gates/biases/Adam_1:0
  scope//rnn/gru_cell/candidate/weights/Adam:0
  scope//rnn/gru_cell/candidate/weights/Adam_1:0
  scope//rnn/gru_cell/candidate/biases/Adam:0
  scope//rnn/gru_cell/candidate/biases/Adam_1:0

optimizer IS scoped, using ORIGINAL scope, variables created with tf.Variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/beta1_power:0
  scope/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer IS scoped, using ORIGINAL scope, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/beta1_power:0
  scope/beta2_power:0
  scope/scope/w/Adam:0
  scope/scope/w/Adam_1:0
  scope/scope/rnn/gru_cell/gates/weights/Adam:0
  scope/scope/rnn/gru_cell/gates/weights/Adam_1:0
  scope/scope/rnn/gru_cell/gates/biases/Adam:0
  scope/scope/rnn/gru_cell/gates/biases/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam:0
  scope/scope/rnn/gru_cell/candidate/weights/Adam_1:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam:0
  scope/scope/rnn/gru_cell/candidate/biases/Adam_1:0

optimizer NOT scoped, re-entering scope with OBJECT, variables created with tf.get_variable
  scope/w:0
  scope/rnn/gru_cell/gates/weights:0
  scope/rnn/gru_cell/gates/biases:0
  scope/rnn/gru_cell/candidate/weights:0
  scope/rnn/gru_cell/candidate/biases:0
  scope/w/RMSProp:0
  scope/w/RMSProp_1:0
  scope/rnn/gru_cell/gates/weights/RMSProp:0
  scope/rnn/gru_cell/gates/weights/RMSProp_1:0
  scope/rnn/gru_cell/gates/biases/RMSProp:0
  scope/rnn/gru_cell/gates/biases/RMSProp_1:0
  scope/rnn/gru_cell/candidate/weights/RMSProp:0
  scope/rnn/gru_cell/candidate/weights/RMSProp_1:0
  scope/rnn/gru_cell/candidate/biases/RMSProp:0
  scope/rnn/gru_cell/candidate/biases/RMSProp_1:0
</code></pre></div>
<p dir="auto">Workaround for clean scoping is to create optimizer op outside of the variable scope and using eg. <code class="notranslate">RMSPropOptimizer</code>. However this is confusing behaviour, ideally everything should be able to be scoped. Unfortunately there is nothing to get variable names clean when using <code class="notranslate">AdamOptimizer</code>. <code class="notranslate">AdamOptimizer</code> creates variables for beta powers with <code class="notranslate">tf.Variable</code>.</p>
<p dir="auto">All of the mentioned problems would probably be fixed by using <code class="notranslate">tf.get_variable</code> everywhere. Until (and if) this change happens users should be instructed to use <code class="notranslate">tf.get_variable</code> only, creating optimizer outside of variable scope and avoid using <code class="notranslate">AdamOptimizer</code> if clean scoping is desired. Double scoping of RNN gradient variables would be tolerable but having several nested classes implementing different models with their own variable scopes lead to extremely messy variable names making debugging more difficult.</p>
<p dir="auto">Tested on Ubuntu 16.04 with Tensorflow 1.0.0 installed with pip.</p>
<p dir="auto">Related issues:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="191018655" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/5786" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/5786/hovercard" href="https://github.com/tensorflow/tensorflow/issues/5786">#5786</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="194303257" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/6189" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/6189/hovercard" href="https://github.com/tensorflow/tensorflow/issues/6189">#6189</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="192800636" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/6007" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/6007/hovercard" href="https://github.com/tensorflow/tensorflow/issues/6007">#6007</a></p>