<ul dir="auto">
<li><strong>OS Platform and Distribution</strong>: Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: binary</li>
<li><strong>TensorFlow version</strong>: 1.4.1</li>
<li><strong>Python version</strong>: 3.5.2</li>
<li><strong>CUDA/cuDNN version</strong>: CUDA 8.0, CUDNN 7.0.5</li>
<li><strong>GPU model and memory</strong>: GTX1080 (8G)</li>
<li><strong>Exact command to reproduce</strong>: python3 test.py</li>
<li><strong>Have I written custom code</strong>: True</li>
<li><strong>Bazel version</strong>: N/A</li>
<li><strong>GCC version</strong>: N/A</li>
</ul>
<p dir="auto">Repoduce code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf

_BATCH_NORM_DECAY = 0.997
_BATCH_NORM_EPSILON = 1e-5

def two_batchnorm(inputs):
    with tf.variable_scope('two_batchnorm'):
        inputs = tf.layers.batch_normalization(
            inputs=inputs,
            axis=3,
            momentum=_BATCH_NORM_DECAY,
            epsilon=_BATCH_NORM_EPSILON,
            center=True,
            scale=True,
            training=True,
            fused=True)
        inputs = tf.layers.batch_normalization(
            inputs=inputs,
            axis=3,
            momentum=_BATCH_NORM_DECAY,
            epsilon=_BATCH_NORM_EPSILON,
            center=True,
            scale=True,
            training=True,
            fused=True)
    return inputs

inputs = tf.placeholder(tf.float32, [1, 5, 5, 3])
x = inputs
x = two_batchnorm(x)
x = two_batchnorm(x)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>

<span class="pl-s1">_BATCH_NORM_DECAY</span> <span class="pl-c1">=</span> <span class="pl-c1">0.997</span>
<span class="pl-s1">_BATCH_NORM_EPSILON</span> <span class="pl-c1">=</span> <span class="pl-c1">1e-5</span>

<span class="pl-k">def</span> <span class="pl-en">two_batchnorm</span>(<span class="pl-s1">inputs</span>):
    <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s">'two_batchnorm'</span>):
        <span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">layers</span>.<span class="pl-en">batch_normalization</span>(
            <span class="pl-s1">inputs</span><span class="pl-c1">=</span><span class="pl-s1">inputs</span>,
            <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">3</span>,
            <span class="pl-s1">momentum</span><span class="pl-c1">=</span><span class="pl-s1">_BATCH_NORM_DECAY</span>,
            <span class="pl-s1">epsilon</span><span class="pl-c1">=</span><span class="pl-s1">_BATCH_NORM_EPSILON</span>,
            <span class="pl-s1">center</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">scale</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">training</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">fused</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
        <span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">layers</span>.<span class="pl-en">batch_normalization</span>(
            <span class="pl-s1">inputs</span><span class="pl-c1">=</span><span class="pl-s1">inputs</span>,
            <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">3</span>,
            <span class="pl-s1">momentum</span><span class="pl-c1">=</span><span class="pl-s1">_BATCH_NORM_DECAY</span>,
            <span class="pl-s1">epsilon</span><span class="pl-c1">=</span><span class="pl-s1">_BATCH_NORM_EPSILON</span>,
            <span class="pl-s1">center</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">scale</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">training</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">fused</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-k">return</span> <span class="pl-s1">inputs</span>

<span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>, [<span class="pl-c1">1</span>, <span class="pl-c1">5</span>, <span class="pl-c1">5</span>, <span class="pl-c1">3</span>])
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">inputs</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-en">two_batchnorm</span>(<span class="pl-s1">x</span>)
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-en">two_batchnorm</span>(<span class="pl-s1">x</span>)</pre></div>
<p dir="auto">It'll trigger an unexpected ValueError as following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError: Variable two_batchnorm/batch_normalization/gamma already exists, disallowed."><pre class="notranslate"><code class="notranslate">ValueError: Variable two_batchnorm/batch_normalization/gamma already exists, disallowed.
</code></pre></div>
<p dir="auto">Removing the variable scope <code class="notranslate">with tf.variable_scope('two_batchnorm')</code> in <code class="notranslate">two_batchnorm</code> will work as expected.</p>
<p dir="auto">All variables defined in the graph should be (in creation sequense):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization_2/gamma:0
two_batchnorm/batch_normalization_2/beta:0
two_batchnorm/batch_normalization_2/moving_mean:0
two_batchnorm/batch_normalization_2/moving_variance:0
two_batchnorm/batch_normalization_3/gamma:0
two_batchnorm/batch_normalization_3/beta:0
two_batchnorm/batch_normalization_3/moving_mean:0
two_batchnorm/batch_normalization_3/moving_variance:0"><pre class="notranslate"><code class="notranslate">two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization_2/gamma:0
two_batchnorm/batch_normalization_2/beta:0
two_batchnorm/batch_normalization_2/moving_mean:0
two_batchnorm/batch_normalization_2/moving_variance:0
two_batchnorm/batch_normalization_3/gamma:0
two_batchnorm/batch_normalization_3/beta:0
two_batchnorm/batch_normalization_3/moving_mean:0
two_batchnorm/batch_normalization_3/moving_variance:0
</code></pre></div>
<p dir="auto">However, with <code class="notranslate">tf.layers.Layer</code>'s <code class="notranslate">add_variable</code> logics, it'll result in an unexpected value name as following (in creation sequence):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization/gamma:0                        &lt;-- ValueError raised here.
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0"><pre class="notranslate"><code class="notranslate">two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization/gamma:0                        &lt;-- ValueError raised here.
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
</code></pre></div>
<p dir="auto">One solution might be using <code class="notranslate">self._name</code> to setup Layer's scope, not <code class="notranslate">self._base_name</code>.</p>
<p dir="auto">I am using tf.TFRecordReader and tf.train.batch to evaluate one epoch of my dataset (it has only one file). In order to get a better performance I tried to use enqueue_many passing a list of reader calls, because supposedly it launches several reader threads. However its behaviour has not been as I expected. It returns each item duplicated, although I thought it would return the same list of items, but faster. I took the code from <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yaroslavvb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yaroslavvb">@yaroslavvb</a> github: <a href="https://github.com/yaroslavvb/stuff/blob/master/ericyue-slowreader/benchmark.py">https://github.com/yaroslavvb/stuff/blob/master/ericyue-slowreader/benchmark.py</a></p>
<p dir="auto">Is this behaviour right? It seems really strange to me and I don't know if it is a bug or a feature.</p>
<p dir="auto">Here you can see a snippet of the code:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="reader = tf.TFRecordReader()
queue_batch = []
for i in range(enqueue_many_size):
    _, serialized_example = reader.read(filename_queue)
    queue_batch.append(serialized_example)
batch_serialized_example = tf.train.batch(
    [queue_batch],
    batch_size=batch_size,
    num_threads=thread_number,
    capacity=capacity,
    enqueue_many=True)"><pre class="notranslate"><code class="notranslate">reader = tf.TFRecordReader()
queue_batch = []
for i in range(enqueue_many_size):
    _, serialized_example = reader.read(filename_queue)
    queue_batch.append(serialized_example)
batch_serialized_example = tf.train.batch(
    [queue_batch],
    batch_size=batch_size,
    num_threads=thread_number,
    capacity=capacity,
    enqueue_many=True)
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu 16.04<br>
Nvidia Geforce 1080<br>
Python 3.5<br>
Tensorflow version: 1.0.0</p>
<p dir="auto">Installed version of CUDA and cuDNN:<br>
/usr/local/cuda-8.0/lib64/libcudadevrt.a       /usr/local/cuda-8.0/lib64/libcudnn.so<br>
/usr/local/cuda-8.0/lib64/libcudart.so         /usr/local/cuda-8.0/lib64/libcudnn.so.5<br>
/usr/local/cuda-8.0/lib64/libcudart.so.8.0     /usr/local/cuda-8.0/lib64/libcudnn.so.5.0.5<br>
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.44  /usr/local/cuda-8.0/lib64/libcudnn.so.5.1.5<br>
/usr/local/cuda-8.0/lib64/libcudart_static.a   /usr/local/cuda-8.0/lib64/libcudnn_static.a</p>
<p dir="auto">The output for batch size of 4 records, enqueue_many_size=2, and record size of 16 floats, is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[array([[  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
     11.,  12.,  13.,  14.,  15.],
   [  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
     11.,  12.,  13.,  14.,  15.],
   [ 16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,
     27.,  28.,  29.,  30.,  31.],
   [ 16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,
     27.,  28.,  29.,  30.,  31.]], dtype=float32)]"><pre class="notranslate"><code class="notranslate">[array([[  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
     11.,  12.,  13.,  14.,  15.],
   [  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
     11.,  12.,  13.,  14.,  15.],
   [ 16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,
     27.,  28.,  29.,  30.,  31.],
   [ 16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,
     27.,  28.,  29.,  30.,  31.]], dtype=float32)]
</code></pre></div>