<p dir="auto">I'm using a LeNet-5 mnist example from Udacity's course. Link to the source code is below.<br>
Training works ok on a CPU (config = tf.ConfigProto(device_count = {'GPU': 0})),<br>
but fails in a GPU mode with the following 'CUDA_ERROR_ILLEGAL_ADDRESS' error:</p>
<blockquote>
<p dir="auto">I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:885] Found device 0 with properties:<br>
name: GeForce GTX 1060 6GB<br>
major: 6 minor: 1 memoryClockRate (GHz) 1.7845<br>
pciBusID 0000:01:00.0<br>
Total memory: 6.00GiB<br>
Free memory: 5.01GiB<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:906] DMA: 0<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:916] 0:   Y<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)<br>
Training...<br>
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: <strong>CUDA_ERROR_ILLEGAL_ADDRESS</strong><br>
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_event_mgr.cc:198] Unexpected Event status: 1</p>
</blockquote>
<p dir="auto">I have reproduced same error on two setups.<br>
Environment 1 (Home PC):</p>
<ul dir="auto">
<li>windows 10;</li>
<li>latest anaconda 4.2.0, python 3.5;</li>
<li>cuda 8</li>
<li>cudnn 5.1 (for win10)</li>
<li>tensorflow 0.12.0 gpu (<a href="https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl</a>)</li>
<li>GeForce GTX 1060 6Gb</li>
</ul>
<p dir="auto">Environment 2 (Work PC):</p>
<ul dir="auto">
<li>windows 7;</li>
<li>latest anaconda 4.2.0, python 3.5;</li>
<li>cuda 8</li>
<li>cudnn 5.1 (for win7)</li>
<li>tensorflow 0.12.0 gpu (<a href="https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl</a>)</li>
<li>GeForce GTX 660 3Gb</li>
</ul>
<p dir="auto">I'm sharing two scripts with minor changes that allow a workaround:<br>
<a href="https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk" rel="nofollow">https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk</a></p>
<p dir="auto">Main difference:<br>
Script from example that crashes (LabLenetBad.py) uses raw mnist label data with the tf.one_hot() call.<br>
The workaround (LabLenetGood.py) reads mnist data with (one_hot=True) flag and does not use tf.one_hot() call on the Y placeholder.</p>
<p dir="auto">I think that tf.one_hot does not work properly on the gpu.</p>
<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  Linux Ubuntu 16.04</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 1.13.1</li>
<li>Python version: 3.5.2</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version: CUDA 10.0</li>
<li>GPU model and memory:</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
Using tf.summary.scalar in keras loss/metrics function, then use model_to_estimator to convert keras model to estimator, tensorboard get duplicate scalars with totally same value. (e.g., <code class="notranslate">metrics/accuracy/train</code> and <code class="notranslate">metrics/accuracy/train_1</code>)</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
It should only summary one scalar.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
A example here:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    ...

    model = tf.keras.Model(inputs=input_layer, outputs=[output_layer1, output_layer2])

    def accuracy(y_true, y_pred):
        y_pred, y_true = reshape_keras_tensor(y_pred, y_true)
        acc = tf.reduce_mean(
            tf.cast(tf.equal(tf.cast(y_true, tf.int64), tf.argmax(input=y_pred, axis=1)), tf.float32))
        # Where I summary, without this line, it will not summary anything during training
        tf.summary.scalar('train', acc)
        return acc

    def total_loss(y_true, y_pred):
        y_pred, y_true = reshape_keras_tensor(y_pred, y_true)
        loss =  tf.nn.sparse_softmax_cross_entropy_with_logits(
            labels=y_true,
            logits=y_pred)
        loss = tf.reduce_mean(cross_entropy)
        # Where I summary, without this line, it will not summary anything during training
        tf.summary.scalar('train', loss)
        return loss

    ...

    model.compile(optimizer=optimizer_builder.build(config.optimizer),
                  loss={output_layer1: total_loss},
                  metrics={output_layer1: accuracy})

    estimator = tf.keras.estimator.model_to_estimator(
        keras_model=model,
        config=run_config,
    )
    train_spec = tf.estimator.TrainSpec(
        input_fn=lambda: train_input_fn(config))
    eval_spec = tf.estimator.EvalSpec(
        input_fn=lambda: eval_input_fn(config))
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"><pre class="notranslate">    ...

    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">keras</span>.<span class="pl-v">Model</span>(<span class="pl-s1">inputs</span><span class="pl-c1">=</span><span class="pl-s1">input_layer</span>, <span class="pl-s1">outputs</span><span class="pl-c1">=</span>[<span class="pl-s1">output_layer1</span>, <span class="pl-s1">output_layer2</span>])

    <span class="pl-k">def</span> <span class="pl-en">accuracy</span>(<span class="pl-s1">y_true</span>, <span class="pl-s1">y_pred</span>):
        <span class="pl-s1">y_pred</span>, <span class="pl-s1">y_true</span> <span class="pl-c1">=</span> <span class="pl-en">reshape_keras_tensor</span>(<span class="pl-s1">y_pred</span>, <span class="pl-s1">y_true</span>)
        <span class="pl-s1">acc</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(
            <span class="pl-s1">tf</span>.<span class="pl-en">cast</span>(<span class="pl-s1">tf</span>.<span class="pl-en">equal</span>(<span class="pl-s1">tf</span>.<span class="pl-en">cast</span>(<span class="pl-s1">y_true</span>, <span class="pl-s1">tf</span>.<span class="pl-s1">int64</span>), <span class="pl-s1">tf</span>.<span class="pl-en">argmax</span>(<span class="pl-s1">input</span><span class="pl-c1">=</span><span class="pl-s1">y_pred</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)), <span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>))
        <span class="pl-c"># Where I summary, without this line, it will not summary anything during training</span>
        <span class="pl-s1">tf</span>.<span class="pl-s1">summary</span>.<span class="pl-en">scalar</span>(<span class="pl-s">'train'</span>, <span class="pl-s1">acc</span>)
        <span class="pl-k">return</span> <span class="pl-s1">acc</span>

    <span class="pl-k">def</span> <span class="pl-en">total_loss</span>(<span class="pl-s1">y_true</span>, <span class="pl-s1">y_pred</span>):
        <span class="pl-s1">y_pred</span>, <span class="pl-s1">y_true</span> <span class="pl-c1">=</span> <span class="pl-en">reshape_keras_tensor</span>(<span class="pl-s1">y_pred</span>, <span class="pl-s1">y_true</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span>  <span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">sparse_softmax_cross_entropy_with_logits</span>(
            <span class="pl-s1">labels</span><span class="pl-c1">=</span><span class="pl-s1">y_true</span>,
            <span class="pl-s1">logits</span><span class="pl-c1">=</span><span class="pl-s1">y_pred</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(<span class="pl-s1">cross_entropy</span>)
        <span class="pl-c"># Where I summary, without this line, it will not summary anything during training</span>
        <span class="pl-s1">tf</span>.<span class="pl-s1">summary</span>.<span class="pl-en">scalar</span>(<span class="pl-s">'train'</span>, <span class="pl-s1">loss</span>)
        <span class="pl-k">return</span> <span class="pl-s1">loss</span>

    ...

    <span class="pl-s1">model</span>.<span class="pl-en">compile</span>(<span class="pl-s1">optimizer</span><span class="pl-c1">=</span><span class="pl-s1">optimizer_builder</span>.<span class="pl-en">build</span>(<span class="pl-s1">config</span>.<span class="pl-s1">optimizer</span>),
                  <span class="pl-s1">loss</span><span class="pl-c1">=</span>{<span class="pl-s1">output_layer1</span>: <span class="pl-s1">total_loss</span>},
                  <span class="pl-s1">metrics</span><span class="pl-c1">=</span>{<span class="pl-s1">output_layer1</span>: <span class="pl-s1">accuracy</span>})

    <span class="pl-s1">estimator</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">keras</span>.<span class="pl-s1">estimator</span>.<span class="pl-en">model_to_estimator</span>(
        <span class="pl-s1">keras_model</span><span class="pl-c1">=</span><span class="pl-s1">model</span>,
        <span class="pl-s1">config</span><span class="pl-c1">=</span><span class="pl-s1">run_config</span>,
    )
    <span class="pl-s1">train_spec</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">estimator</span>.<span class="pl-v">TrainSpec</span>(
        <span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>: <span class="pl-en">train_input_fn</span>(<span class="pl-s1">config</span>))
    <span class="pl-s1">eval_spec</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">estimator</span>.<span class="pl-v">EvalSpec</span>(
        <span class="pl-s1">input_fn</span><span class="pl-c1">=</span><span class="pl-k">lambda</span>: <span class="pl-en">eval_input_fn</span>(<span class="pl-s1">config</span>))
    <span class="pl-s1">tf</span>.<span class="pl-s1">estimator</span>.<span class="pl-en">train_and_evaluate</span>(<span class="pl-s1">estimator</span>, <span class="pl-s1">train_spec</span>, <span class="pl-s1">eval_spec</span>)</pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
TensorBoard with duplicate scalars:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/7609173/65102919-27da0c00-d9ff-11e9-8a46-6d00ea39a4eb.png"><img src="https://user-images.githubusercontent.com/7609173/65102919-27da0c00-d9ff-11e9-8a46-6d00ea39a4eb.png" alt="2019-09-18 10 28 17" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/7609173/65102920-2872a280-d9ff-11e9-9d85-b626593688a7.png"><img src="https://user-images.githubusercontent.com/7609173/65102920-2872a280-d9ff-11e9-9d85-b626593688a7.png" alt="2019-09-18 10 27 56" style="max-width: 100%;"></a></p>