<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="136722135" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/1300" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/1300/hovercard" href="https://github.com/tensorflow/tensorflow/issues/1300">#1300</a></p>
<p dir="auto">I have got a nvidia p100 GPU which is support fp16, and I run the TF case 'cifar10_train.py'. Without option '--use_fp16', the performance is also 1600 examples/sec, and with the option '--use_fp16', the performance down to 500 examples/sec. Any ideas about this issue?</p>
<p dir="auto">userid@ubuntu-WK-4xP100:<del>/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ vi cifar10_train.py<br>
userid@ubuntu-WK-4xP100:</del>/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally</p>
<blockquote>
<blockquote>
<p dir="auto">Downloading cifar-10-binary.tar.gz 100.0%<br>
...<br>
2016-11-14 00:07:57.143739: step 0, loss = 4.67 (12.1 examples/sec; 10.549 sec/batch)<br>
2016-11-14 00:07:58.395209: step 10, loss = 4.57 (1693.6 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:07:59.177525: step 20, loss = 4.97 (1668.9 examples/sec; 0.077 sec/batch)<br>
2016-11-14 00:07:59.957789: step 30, loss = 4.43 (1588.3 examples/sec; 0.081 sec/batch)<br>
2016-11-14 00:08:00.738431: step 40, loss = 4.52 (1690.5 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:08:01.501940: step 50, loss = 4.33 (1680.4 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:08:02.241604: step 60, loss = 4.20 (1733.4 examples/sec; 0.074 sec/batch)<br>
2016-11-14 00:08:03.001845: step 70, loss = 4.27 (1706.0 examples/sec; 0.075 sec/batch)<br>
2016-11-14 00:08:03.765522: step 80, loss = 4.18 (1601.3 examples/sec; 0.080 sec/batch)<br>
2016-11-14 00:08:04.516780: step 90, loss = 4.25 (1646.3 examples/sec; 0.078 sec/batch)</p>
</blockquote>
</blockquote>
<p dir="auto">userid@ubuntu-WK-4xP100:~/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py --use_fp16<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally<br>
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.<br>
..<br>
2016-11-14 00:09:09.382854: step 0, loss = 4.67 (12.3 examples/sec; 10.411 sec/batch)<br>
2016-11-14 00:09:11.923842: step 10, loss = 4.58 (659.6 examples/sec; 0.194 sec/batch)<br>
2016-11-14 00:09:13.918448: step 20, loss = 6.62 (460.6 examples/sec; 0.278 sec/batch)<br>
2016-11-14 00:09:16.211809: step 30, loss = 4.37 (583.7 examples/sec; 0.219 sec/batch)<br>
2016-11-14 00:09:18.327690: step 40, loss = 4.30 (618.3 examples/sec; 0.207 sec/batch)<br>
2016-11-14 00:09:20.395409: step 50, loss = 4.37 (643.8 examples/sec; 0.199 sec/batch)<br>
2016-11-14 00:09:22.466230: step 60, loss = 4.32 (574.0 examples/sec; 0.223 sec/batch)<br>
2016-11-14 00:09:24.533225: step 70, loss = 4.17 (646.2 examples/sec; 0.198 sec/batch)<br>
2016-11-14 00:09:26.609277: step 80, loss = 2.59 (601.5 examples/sec; 0.213 sec/batch)<br>
2016-11-14 00:09:28.703648: step 90, loss = 4.15 (625.5 examples/sec; 0.205 sec/batch)</p>
<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS High Sierra</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 1.13.1</li>
<li>Python version: 2.7.14</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version: None</li>
<li>GPU model and memory: None</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture <a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with<br>
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">I'm currently using MonitoredTrainingSession and passing it the CheckpointSaverHook. It works fine for new models. When I want to restore a model, I give a path to the <code class="notranslate">checkpoint_dir</code> arg. However, this also inits the default CheckpointSaver, and it will double save checkpoints in the directory. This is evident if I change <code class="notranslate">checkpoint_basename</code> as I get duplicate checkpoints at each global step.</p>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">I would not expect the default CheckpointSaver to init, and instead use the params given by the CheckpointSaverHook.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
    initializer_hook = _DatasetInitializerHook(init_train, self.train_tfrecords_glob,
                                               queue_name=self.tensorq_name)
    summary_hook = tf.train.SummarySaverHook(save_steps=50, save_secs=None, summary_op=summary_op)
    logging_hook =  tf.train.LoggingTensorHook(tensors={'step': global_step, 'loss': loss,
                                                        'accuracy': acc_op}, 
                                                         every_n_iter=logging_step)
    saver = tf.train.Saver(max_to_keep=2)
    saver_hook = tf.train.CheckpointSaverHook(checkpoint_dir=self.ckpt_dir, 
                                              saver=saver, save_steps=save_step,
                                              checkpoint_basename='latest_model')
    step_hook = tf.train.StepCounterHook(every_n_steps=logging_step)

    hooks = [summary_hook, logging_hook]

    # Checkpoint Saver - Check for new model
    if self.new_model:
      logging.info('Starting New Model...existing ckpts will be deleted')
      hooks.extend([saver_hook, step_hook])
      fl = [f for f in os.listdir(self.ckpt_dir)]
      for f in fl:
        os.remove(os.path.join(self.ckpt_dir, f))
      checkpoint_dir =  None

    else:
      checkpoint_dir = self.ckpt_dir
      hooks.append(saver_hook)
    
    # Start Training
    with tf.train.MonitoredTrainingSession(
        checkpoint_dir=checkpoint_dir,
        hooks=[initializer_hook],
        chief_only_hooks=hooks,
        log_step_count_steps=logging_step,
        config=tf.ConfigProto(allow_soft_placement=True)) as mon_sess:
        while not mon_sess.should_stop():
          mon_sess.run(training_op)

    logging.info(&quot;Training Completed&quot;)"><pre class="notranslate"><code class="notranslate">...
    initializer_hook = _DatasetInitializerHook(init_train, self.train_tfrecords_glob,
                                               queue_name=self.tensorq_name)
    summary_hook = tf.train.SummarySaverHook(save_steps=50, save_secs=None, summary_op=summary_op)
    logging_hook =  tf.train.LoggingTensorHook(tensors={'step': global_step, 'loss': loss,
                                                        'accuracy': acc_op}, 
                                                         every_n_iter=logging_step)
    saver = tf.train.Saver(max_to_keep=2)
    saver_hook = tf.train.CheckpointSaverHook(checkpoint_dir=self.ckpt_dir, 
                                              saver=saver, save_steps=save_step,
                                              checkpoint_basename='latest_model')
    step_hook = tf.train.StepCounterHook(every_n_steps=logging_step)

    hooks = [summary_hook, logging_hook]

    # Checkpoint Saver - Check for new model
    if self.new_model:
      logging.info('Starting New Model...existing ckpts will be deleted')
      hooks.extend([saver_hook, step_hook])
      fl = [f for f in os.listdir(self.ckpt_dir)]
      for f in fl:
        os.remove(os.path.join(self.ckpt_dir, f))
      checkpoint_dir =  None

    else:
      checkpoint_dir = self.ckpt_dir
      hooks.append(saver_hook)
    
    # Start Training
    with tf.train.MonitoredTrainingSession(
        checkpoint_dir=checkpoint_dir,
        hooks=[initializer_hook],
        chief_only_hooks=hooks,
        log_step_count_steps=logging_step,
        config=tf.ConfigProto(allow_soft_placement=True)) as mon_sess:
        while not mon_sess.should_stop():
          mon_sess.run(training_op)

    logging.info("Training Completed")
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>