<p dir="auto">Please go to Stack Overflow for help and support:</p>
<p dir="auto"><a href="https://stackoverflow.com/questions/tagged/tensorflow" rel="nofollow">https://stackoverflow.com/questions/tagged/tensorflow</a></p>
<p dir="auto">If you open a GitHub issue, here is our policy:</p>
<ol dir="auto">
<li>It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).</li>
<li>The form below must be filled out.</li>
<li>It shouldn't be a TensorBoard issue. Those go <a href="https://github.com/tensorflow/tensorboard/issues">here</a>.</li>
</ol>
<p dir="auto"><strong>Here's why we have that policy</strong>: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.</p>
<hr>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>:</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:</li>
<li><strong>TensorFlow version (use command below)</strong>:</li>
<li><strong>Python version</strong>:</li>
<li><strong>Bazel version (if compiling from source)</strong>:</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:</li>
<li><strong>CUDA/cuDNN version</strong>:</li>
<li><strong>GPU model and memory</strong>:</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture script:</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh</a></p>
<p dir="auto">You can obtain the TensorFlow version with</p>
<p dir="auto">python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.</p>
<p dir="auto">I want to train with multiple gpu with tensorflow-1.10. I use the estimator and MirroredStrategy. The model is resnet50 which is official model of tensorflow . The following is my code and error log. Am I use MirroredStrategy in the wrong way?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np
from tensorflow.contrib.slim.nets import resnet_v2
import tensorflow.contrib.slim as slim

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data(path='mnist.npz')
x_train = np.expand_dims(x_train, 3).astype(np.float32)
y_train = y_train.astype(np.int32)
x_test = np.expand_dims(x_test, 3).astype(np.float32)
y_test = y_test.astype(np.int32)

tf.logging.set_verbosity(tf.logging.INFO)

cls_num = 10


def model_fn(features, labels, mode):
    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
    with slim.arg_scope(resnet_v2.resnet_arg_scope()):
        logits, endpoints = resnet_v2.resnet_v2_50(features, 
                num_classes=cls_num,
                is_training=is_training)

    logits = tf.squeeze(logits, [1, 2])
    preds = tf.argmax(logits, 1)
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
    optimizer = tf.train.AdagradOptimizer(learning_rate=0.01)
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())

    accuracy = tf.metrics.accuracy(labels=labels, predictions=preds)
    metrics = {'accuracy': accuracy}

    if mode == tf.estimator.ModeKeys.EVAL:
        return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)

    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op) 


def train_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
    dataset = dataset.repeat(1).batch(128)
    return dataset

def eval_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
    dataset = dataset.repeat(1).batch(128)
    return dataset


distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=distribution)
estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir='logs', config=config)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn)
eval_specs = tf.estimator.EvalSpec(input_fn=eval_input_fn)
for _ in xrange(100):
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import numpy as np
from tensorflow.contrib.slim.nets import resnet_v2
import tensorflow.contrib.slim as slim

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data(path='mnist.npz')
x_train = np.expand_dims(x_train, 3).astype(np.float32)
y_train = y_train.astype(np.int32)
x_test = np.expand_dims(x_test, 3).astype(np.float32)
y_test = y_test.astype(np.int32)

tf.logging.set_verbosity(tf.logging.INFO)

cls_num = 10


def model_fn(features, labels, mode):
    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
    with slim.arg_scope(resnet_v2.resnet_arg_scope()):
        logits, endpoints = resnet_v2.resnet_v2_50(features, 
                num_classes=cls_num,
                is_training=is_training)

    logits = tf.squeeze(logits, [1, 2])
    preds = tf.argmax(logits, 1)
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
    optimizer = tf.train.AdagradOptimizer(learning_rate=0.01)
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())

    accuracy = tf.metrics.accuracy(labels=labels, predictions=preds)
    metrics = {'accuracy': accuracy}

    if mode == tf.estimator.ModeKeys.EVAL:
        return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)

    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op) 


def train_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
    dataset = dataset.repeat(1).batch(128)
    return dataset

def eval_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
    dataset = dataset.repeat(1).batch(128)
    return dataset


distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=distribution)
estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir='logs', config=config)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn)
eval_specs = tf.estimator.EvalSpec(input_fn=eval_input_fn)
for _ in xrange(100):
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)
</code></pre></div>
<p dir="auto">This is the error log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-08-30 16:58:47.446417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 
2018-08-30 16:58:47.446428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y 
2018-08-30 16:58:47.446437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N 
2018-08-30 16:58:47.446877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 21544 MB memory) -&gt; physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
2018-08-30 16:58:47.447098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 21544 MB memory) -&gt; physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
INFO:tensorflow:Device is available but not used by distribute strategy: /device:CPU:0
INFO:tensorflow:Configured nccl all-reduce.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Error reported to Coordinator: 
Traceback (most recent call last):
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py&quot;, line 297, in stop_on_exception
    yield
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py&quot;, line 519, in run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py&quot;, line 1133, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File &quot;test_mnist.py&quot;, line 22, in model_fn
    is_training=is_training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 287, in resnet_v2_50
    scope=scope)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 216, in resnet_v2
    net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py&quot;, line 215, in stack_blocks_dense
    net = block.unit_fn(net, rate=1, **unit)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 101, in bottleneck
    inputs, activation_fn=nn_ops.relu, scope='preact')
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py&quot;, line 650, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py&quot;, line 805, in apply
    return self.__call__(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py&quot;, line 362, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py&quot;, line 736, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py&quot;, line 158, in call
    return super(BatchNormalization, self).call(inputs, training=training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 514, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 420, in _fused_batch_norm
    momentum)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 369, in _assign_moving_average
    with ops.colocate_with(variable):
  File &quot;/home/soft/lib/python2.7/contextlib.py&quot;, line 17, in __enter__
    return self.gen.next()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 3939, in _colocate_with_for_gradient
    with self.colocate_with(op, ignore_existing):
  File &quot;/home/soft/lib/python2.7/contextlib.py&quot;, line 17, in __enter__
    return self.gen.next()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 3992, in colocate_with
    op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 1255, in internal_convert_to_tensor_or_indexed_slices
    value, dtype=dtype, name=name, as_ref=as_ref)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 1094, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py&quot;, line 414, in _tensor_conversion_mirrored
    assert not as_ref
AssertionError
Traceback (most recent call last):
  File &quot;test_mnist.py&quot;, line 58, in &lt;module&gt;
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py&quot;, line 451, in train_and_evaluate
    return executor.run()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py&quot;, line 590, in run
    return self.run_local()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py&quot;, line 691, in run_local
    saving_listeners=saving_listeners)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py&quot;, line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py&quot;, line 1143, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py&quot;, line 1255, in _train_model_distributed
    self.config)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/training/distribute.py&quot;, line 777, in call_for_each_tower
    return self._call_for_each_tower(fn, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py&quot;, line 308, in _call_for_each_tower
    coord.join(threads)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py&quot;, line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py&quot;, line 297, in stop_on_exception
    yield
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py&quot;, line 519, in run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py&quot;, line 1133, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File &quot;test_mnist.py&quot;, line 22, in model_fn
    is_training=is_training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 287, in resnet_v2_50
    scope=scope)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 216, in resnet_v2
    net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py&quot;, line 215, in stack_blocks_dense
    net = block.unit_fn(net, rate=1, **unit)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py&quot;, line 101, in bottleneck
    inputs, activation_fn=nn_ops.relu, scope='preact')
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py&quot;, line 183, in func_with_args
    return func(*args, **current_args)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py&quot;, line 650, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py&quot;, line 805, in apply
    return self.__call__(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py&quot;, line 362, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py&quot;, line 736, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py&quot;, line 158, in call
    return super(BatchNormalization, self).call(inputs, training=training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 514, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 420, in _fused_batch_norm
    momentum)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py&quot;, line 369, in _assign_moving_average
    with ops.colocate_with(variable):
  File &quot;/home/soft/lib/python2.7/contextlib.py&quot;, line 17, in __enter__
    return self.gen.next()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 3939, in _colocate_with_for_gradient
    with self.colocate_with(op, ignore_existing):
  File &quot;/home/soft/lib/python2.7/contextlib.py&quot;, line 17, in __enter__
    return self.gen.next()
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 3992, in colocate_with
    op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 1255, in internal_convert_to_tensor_or_indexed_slices
    value, dtype=dtype, name=name, as_ref=as_ref)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py&quot;, line 1094, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File &quot;/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py&quot;, line 414, in _tensor_conversion_mirrored
    assert not as_ref
AssertionError"><pre class="notranslate"><code class="notranslate">2018-08-30 16:58:47.446417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 
2018-08-30 16:58:47.446428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y 
2018-08-30 16:58:47.446437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N 
2018-08-30 16:58:47.446877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 21544 MB memory) -&gt; physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
2018-08-30 16:58:47.447098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 21544 MB memory) -&gt; physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
INFO:tensorflow:Device is available but not used by distribute strategy: /device:CPU:0
INFO:tensorflow:Configured nccl all-reduce.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Error reported to Coordinator: 
Traceback (most recent call last):
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
    yield
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 519, in run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "test_mnist.py", line 22, in model_fn
    is_training=is_training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 287, in resnet_v2_50
    scope=scope)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 216, in resnet_v2
    net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py", line 215, in stack_blocks_dense
    net = block.unit_fn(net, rate=1, **unit)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 101, in bottleneck
    inputs, activation_fn=nn_ops.relu, scope='preact')
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 650, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 805, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 362, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 736, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py", line 158, in call
    return super(BatchNormalization, self).call(inputs, training=training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 514, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 420, in _fused_batch_norm
    momentum)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 369, in _assign_moving_average
    with ops.colocate_with(variable):
  File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3939, in _colocate_with_for_gradient
    with self.colocate_with(op, ignore_existing):
  File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3992, in colocate_with
    op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1255, in internal_convert_to_tensor_or_indexed_slices
    value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py", line 414, in _tensor_conversion_mirrored
    assert not as_ref
AssertionError
Traceback (most recent call last):
  File "test_mnist.py", line 58, in &lt;module&gt;
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
    return executor.run()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 590, in run
    return self.run_local()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 691, in run_local
    saving_listeners=saving_listeners)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1143, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1255, in _train_model_distributed
    self.config)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 777, in call_for_each_tower
    return self._call_for_each_tower(fn, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 308, in _call_for_each_tower
    coord.join(threads)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
    yield
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 519, in run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "test_mnist.py", line 22, in model_fn
    is_training=is_training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 287, in resnet_v2_50
    scope=scope)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 216, in resnet_v2
    net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py", line 215, in stack_blocks_dense
    net = block.unit_fn(net, rate=1, **unit)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 101, in bottleneck
    inputs, activation_fn=nn_ops.relu, scope='preact')
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 650, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 805, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 362, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 736, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py", line 158, in call
    return super(BatchNormalization, self).call(inputs, training=training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 514, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 420, in _fused_batch_norm
    momentum)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 369, in _assign_moving_average
    with ops.colocate_with(variable):
  File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3939, in _colocate_with_for_gradient
    with self.colocate_with(op, ignore_existing):
  File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3992, in colocate_with
    op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1255, in internal_convert_to_tensor_or_indexed_slices
    value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py", line 414, in _tensor_conversion_mirrored
    assert not as_ref
AssertionError
</code></pre></div>