title: Tensorflow automatic mixed precision fp16 slower than fp32 on official resnet
body: <p>I am trying to use the official ResNet model benchmarks from <a href="https://github.com/tensorflow/models/blob/master/official/resnet/estimator_benchmark.py#L191" rel="nofollow noreferrer">https://github.com/tensorflow/models/blob/master/official/resnet/estimator_benchmark.py#L191</a> to experiment with the AMP support included in <code>tensorflow-gpu==1.14.0rc0</code>. I'm running on a 2080 Ti, driver 410.78, CUDA 10, Ubuntu.</p>
  
  <p>I have made the following changes to help make sure the comparisons are quick and apples-to-apples:</p>
  
  <ul>
  <li>Reduced epochs to 10.</li>
  <li>Removed the 2x larger batch size for the <code>tweaked</code> runs so that everything is training on the same number of samples.</li>
  <li>Set the checkpointing to only happen once, after training is finished.</li>
  <li>Switched training to use CIFAR-10, since I have that downloaded on local disk.</li>
  </ul>
  
  <p>I see this in the logs, which implies to me that AMP is active:</p>
  
  <pre><code>2019-06-03 16:08:40.976829: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1767] Running auto_mixed_precision graph optimizer
  2019-06-03 16:08:40.977057: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:40.985402: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:40.986858: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:40.987745: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:40.996781: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:41.001948: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:41.003208: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:41.004589: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:41.005981: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1241] No whitelist ops found, nothing to do
  2019-06-03 16:08:41.511761: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1767] Running auto_mixed_precision graph optimizer
  2019-06-03 16:08:41.527751: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1723] Converted 529/2910 nodes to float16 precision using 3 cast(s) to float16 (excluding Const and Variable casts)
  </code></pre>
  
  <p>But the actual runtime is slower:</p>
  
  <p><a href="https://i.stack.imgur.com/OYYro.png" rel="nofollow noreferrer">fp32 (cyan) runtime is less than all of the fp16 runs.</a></p>
  
  <p>What can I do to see a performance improvement?</p>