<p dir="auto">Hi,<br>
sorry to bother again. In order to save some space I would like to quantize my graph, but everytime I run quantize_graph, it becomes unusable in android app later on. I would like to generally ask, how is this facility meant to be used. What I have done:</p>
<ol dir="auto">
<li>retrained graph (works fine)</li>
<li>optimized it using optimize_for_inference (works fine)</li>
<li>bazel-bin/tensorflow/tools/quantization/quantize_graph <br>
--input=/tmp/optimized.pb <br>
--output=/tmp/eightbit.pb <br>
--output_node_names="final_result_a,final_result_b" <br>
--mode=eightbit</li>
</ol>
<p dir="auto">Then it threws on android following (excerpt):</p>
<p dir="auto"><code class="notranslate">I/native: tensorflow_inference_jni.cc:85 Creating new session variables for 635cce3c13fa1ff4 I/native: tensorflow_inference_jni.cc:113 Loading Tensorflow. I/native: tensorflow_inference_jni.cc:120 Session created. I/native: tensorflow_inference_jni.cc:126 Acquired AssetManager. I/native: tensorflow_inference_jni.cc:128 Reading file to proto: file:///android_asset/eightbit.pb I/native: jni_utils.cc:111 Opening asset eightbit.pb from disk with zero-copy. I/native: tensorflow_inference_jni.cc:132 GraphDef loaded from file:///android_asset/eightbit.pb with 1345 nodes. I/native: stat_summarizer.cc:38 StatSummarizer found 1345 nodes I/native: tensorflow_inference_jni.cc:139 Creating TensorFlow graph from GraphDef. I/native: tensorflow_inference_jni.cc:151 Initialization done in 234.141ms </code><br>
Which I suppose is correct</p>
<p dir="auto">But then:</p>
<p dir="auto"><code class="notranslate">I/native: tensorflow_inference_jni.cc:228 End computing. Ran in 2665ms (2665ms avg over 1 runs) A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0xb95eb000 in tid 31217 (inference) Application terminated.</code><br>
Would you please once again help? Thanks</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Linux Ubuntu 16.04.4 LTS</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: No</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: binary</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.10.0-rc1 and nightly</li>
<li><strong>Python version</strong>: Python 3.5.2</li>
<li><strong>Bazel version (if compiling from source)</strong>: No</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: No</li>
<li><strong>CUDA/cuDNN version</strong>: 9.0.176</li>
<li><strong>GPU model and memory</strong>: Tesla V100 16152MiB</li>
<li><strong>Exact command to reproduce</strong>: Save the snippet in test.py and run "python test.py"</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Using the distribute strategy (OneDeviceStrategy) I have a crash with train_and_evaluate method of the estimator.<br>
The code below works with train and evaluate as separeted functions, but it doesn't work with train_and_evaluate.<br>
I have attached a snipped with train, evaluate and train_and_evaluate methods togheter to gather the behaviour differences. At the end I've put the output error log.<br>
Note: it works with tensorflow 1.9.0 and codalab</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from tensorflow import keras as ks
import numpy as np
import tensorflow as tf
from tensorflow.python.estimator import keras as keras_lib


tf.logging.set_verbosity(tf.logging.INFO)


def input_fn():
    x = np.random.random((1024, 10))
    y = np.random.randint(2, size=(1024, 1))
    x = tf.cast(x, tf.float32)
    dataset = tf.data.Dataset.from_tensor_slices((x, y))
    dataset = dataset.repeat(100)
    dataset = dataset.batch(1)
    return dataset


model = ks.Sequential()
model.add(ks.layers.Dense(16, activation='relu', input_shape=(10,)))
model.add(ks.layers.Dense(1, activation='sigmoid'))

optimizer = tf.train.GradientDescentOptimizer(0.2)

model.compile(loss='binary_crossentropy', optimizer=optimizer)

strategy = tf.contrib.distribute.OneDeviceStrategy(&quot;device:GPU:0&quot;)
config = tf.estimator.RunConfig(train_distribute=strategy)

keras_estimator = keras_lib.model_to_estimator(
  keras_model=model,
  config=config)

keras_estimator.train(input_fn=input_fn, steps=10)
keras_estimator.evaluate(input_fn=input_fn, steps=3)

train_spec = tf.estimator.TrainSpec(
    input_fn=input_fn,
    max_steps=20)
eval_spec = tf.estimator.EvalSpec(
    input_fn=input_fn,
    steps=3)

tf.estimator.train_and_evaluate(keras_estimator, train_spec, eval_spec)"><pre class="notranslate"><code class="notranslate">from tensorflow import keras as ks
import numpy as np
import tensorflow as tf
from tensorflow.python.estimator import keras as keras_lib


tf.logging.set_verbosity(tf.logging.INFO)


def input_fn():
    x = np.random.random((1024, 10))
    y = np.random.randint(2, size=(1024, 1))
    x = tf.cast(x, tf.float32)
    dataset = tf.data.Dataset.from_tensor_slices((x, y))
    dataset = dataset.repeat(100)
    dataset = dataset.batch(1)
    return dataset


model = ks.Sequential()
model.add(ks.layers.Dense(16, activation='relu', input_shape=(10,)))
model.add(ks.layers.Dense(1, activation='sigmoid'))

optimizer = tf.train.GradientDescentOptimizer(0.2)

model.compile(loss='binary_crossentropy', optimizer=optimizer)

strategy = tf.contrib.distribute.OneDeviceStrategy("device:GPU:0")
config = tf.estimator.RunConfig(train_distribute=strategy)

keras_estimator = keras_lib.model_to_estimator(
  keras_model=model,
  config=config)

keras_estimator.train(input_fn=input_fn, steps=10)
keras_estimator.evaluate(input_fn=input_fn, steps=3)

train_spec = tf.estimator.TrainSpec(
    input_fn=input_fn,
    max_steps=20)
eval_spec = tf.estimator.EvalSpec(
    input_fn=input_fn,
    steps=3)

tf.estimator.train_and_evaluate(keras_estimator, train_spec, eval_spec)
</code></pre></div>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INFO:tensorflow:Using the Keras model provided.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp_xh7nsci
INFO:tensorflow:Using config: {'_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_global_id_in_cluster': 0, '_task_type': 'worker', '_session_config': None, '_is_chief': True, '_evaluation_master': '', '_log_step_count_steps': 100, '_model_dir': '/tmp/tmp_xh7nsci', '_save_checkpoints_secs': 600, '_save_summary_steps': 100, '_service': None, '_save_checkpoints_steps': None, '_task_id': 0, '_cluster_spec': &lt;tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9a715b1240&gt;, '_device_fn': None, '_keep_checkpoint_max': 5, '_num_worker_replicas': 1, '_train_distribute': &lt;tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x7f9a630426d8&gt;}
2018-08-06 11:45:02.557498: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-08-06 11:45:02.679737: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-06 11:45:02.680248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1404] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.77GiB freeMemory: 15.36GiB
2018-08-06 11:45:02.680294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:03.032268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:03.032321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:03.032339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:03.032678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-08-06 11:45:03.032984: E tensorflow/core/common_runtime/gpu/gpu_device.cc:228] Illegal GPUOptions.experimental.num_dev_to_dev_copy_streams=0 set to 1 instead.
2018-08-06 11:45:03.353069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:03.353135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:03.353154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:03.353162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:03.353297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:04.042046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:04.042111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:04.042128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:04.042148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:04.042305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/keras_model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:loss = 0.6695193, step = 0
INFO:tensorflow:Saving checkpoints for 10 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:Loss for final step: 0.3536753.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-08-06-11:45:04
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:04.822515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:04.822570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:04.822594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:04.822613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:04.822780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1/3]
INFO:tensorflow:Evaluation [2/3]
INFO:tensorflow:Evaluation [3/3]
INFO:tensorflow:Finished evaluation at 2018-08-06-11:45:04
INFO:tensorflow:Saving dict for global step 10: global_step = 10, loss = 0.1408012
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10: /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:05.220540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:05.220581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:05.220600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:05.220615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:05.220760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 10 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:loss = 1.5840994, step = 10
INFO:tensorflow:Saving checkpoints for 20 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
Traceback (most recent call last):
  File &quot;test2.py&quot;, line 45, in &lt;module&gt;
    tf.estimator.train_and_evaluate(keras_estimator, train_spec, eval_spec)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 451, in train_and_evaluate
    return executor.run()
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 590, in run
    return self.run_local()
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 691, in run_local
    saving_listeners=saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 1143, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 1368, in _train_model_distributed
    saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 1451, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py&quot;, line 695, in __exit__
    self._close_internal(exception_type)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py&quot;, line 727, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/basic_session_run_hooks.py&quot;, line 470, in end
    self._save(session, last_step)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/basic_session_run_hooks.py&quot;, line 489, in _save
    if l.after_save(session, step):
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 497, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 517, in _evaluate
    self._evaluator.evaluate_and_export())
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 884, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 463, in evaluate
    input_fn, hooks, checkpoint_path)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 1474, in _evaluate_build_graph
    model_fn_lib.LOSS_METRIC_KEY] = metrics_lib.mean(estimator_spec.loss)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py&quot;, line 376, in mean
    mean_t = distribute_lib.get_tower_context().merge_call(
AttributeError: 'NoneType' object has no attribute 'merge_call'
"><pre class="notranslate"><code class="notranslate">INFO:tensorflow:Using the Keras model provided.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp_xh7nsci
INFO:tensorflow:Using config: {'_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_global_id_in_cluster': 0, '_task_type': 'worker', '_session_config': None, '_is_chief': True, '_evaluation_master': '', '_log_step_count_steps': 100, '_model_dir': '/tmp/tmp_xh7nsci', '_save_checkpoints_secs': 600, '_save_summary_steps': 100, '_service': None, '_save_checkpoints_steps': None, '_task_id': 0, '_cluster_spec': &lt;tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9a715b1240&gt;, '_device_fn': None, '_keep_checkpoint_max': 5, '_num_worker_replicas': 1, '_train_distribute': &lt;tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x7f9a630426d8&gt;}
2018-08-06 11:45:02.557498: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-08-06 11:45:02.679737: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-06 11:45:02.680248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1404] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.77GiB freeMemory: 15.36GiB
2018-08-06 11:45:02.680294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:03.032268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:03.032321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:03.032339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:03.032678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-08-06 11:45:03.032984: E tensorflow/core/common_runtime/gpu/gpu_device.cc:228] Illegal GPUOptions.experimental.num_dev_to_dev_copy_streams=0 set to 1 instead.
2018-08-06 11:45:03.353069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:03.353135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:03.353154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:03.353162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:03.353297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:04.042046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:04.042111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:04.042128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:04.042148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:04.042305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/keras_model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:loss = 0.6695193, step = 0
INFO:tensorflow:Saving checkpoints for 10 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:Loss for final step: 0.3536753.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-08-06-11:45:04
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:04.822515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:04.822570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:04.822594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:04.822613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:04.822780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1/3]
INFO:tensorflow:Evaluation [2/3]
INFO:tensorflow:Evaluation [3/3]
INFO:tensorflow:Finished evaluation at 2018-08-06-11:45:04
INFO:tensorflow:Saving dict for global step 10: global_step = 10, loss = 0.1408012
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10: /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-08-06 11:45:05.220540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-06 11:45:05.220581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-06 11:45:05.220600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-06 11:45:05.220615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-06 11:45:05.220760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14856 MB memory) -&gt; physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from /tmp/tmp_xh7nsci/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 10 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:loss = 1.5840994, step = 10
INFO:tensorflow:Saving checkpoints for 20 into /tmp/tmp_xh7nsci/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
Traceback (most recent call last):
  File "test2.py", line 45, in &lt;module&gt;
    tf.estimator.train_and_evaluate(keras_estimator, train_spec, eval_spec)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 590, in run
    return self.run_local()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 691, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1143, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1368, in _train_model_distributed
    saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1451, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 695, in __exit__
    self._close_internal(exception_type)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 727, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 470, in end
    self._save(session, last_step)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 489, in _save
    if l.after_save(session, step):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 497, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 517, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 884, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 463, in evaluate
    input_fn, hooks, checkpoint_path)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1474, in _evaluate_build_graph
    model_fn_lib.LOSS_METRIC_KEY] = metrics_lib.mean(estimator_spec.loss)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py", line 376, in mean
    mean_t = distribute_lib.get_tower_context().merge_call(
AttributeError: 'NoneType' object has no attribute 'merge_call'

</code></pre></div>