<p dir="auto">See below. Please let me know what I can do to provide more information for you.  I am working on pulling out the offending code into a standalone py file to replicate the bug elsewhere, but it might take a few days.</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: arch linux (LTS kernel 4.9.44-1)</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: source</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.3 (master branch as of commit <code class="notranslate">566d167c</code>)</li>
<li><strong>Python version</strong>: 3.6.2</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.5.2-2</li>
<li><strong>CUDA/cuDNN version</strong>: 8.0.61-2/6.0.21-2</li>
<li><strong>GPU model and memory</strong>: 1080 GTX Ti</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<p dir="auto">I don't have exact source for you yet of a trimmed down example. I will see if I can put one together. Generally, the steps are</p>
<ul dir="auto">
<li>Generate a few <code class="notranslate">TFRecord</code> files</li>
<li>Instantiate a few <code class="notranslate">TFRecordDataset</code> from the files.</li>
<li>Perform necessary pre-processing on <code class="notranslate">TFRecordDataset</code> entries for each dataset, then on the dataset itself (<code class="notranslate">.cache</code>, <code class="notranslate">.repeat</code>, <code class="notranslate">.shuffle</code>, <code class="notranslate">.batch</code>, etc) For example, we might have datasets <code class="notranslate">train_dataset</code>, <code class="notranslate">validate_dataset</code>, and <code class="notranslate">test_dataset</code>.</li>
<li>Initialize each dataset (<code class="notranslate">.make_initializable_iterator</code>)</li>
<li>Train a model using an input tensor <code class="notranslate">x</code> of</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="train_phase = tf.placeholder(dtype=tf.int32, shape=())
x = tf.case([(tf.equal(train_phase, 1), lambda: train_dataset),
             (tf.equal(train_phase, 2), lambda: validate_dataset),
             (tf.equal(train_phase,3), lambda: test_dataset)],
         default=lambda: train_dataset)
# ... construct model_op using tensor x as input ...
# ... call the initializer ...

# train for a bunch
for i in ...
     sess.run(main_op, feed_dict={train_phase: 1})
# validate
for i in ...
     sess.run(main_op, feed_dict={train_phase: 2})
# test
for i in ...
     sess.run(main_op, feed_dict={train_phase: 3})"><pre class="notranslate"><code class="notranslate">train_phase = tf.placeholder(dtype=tf.int32, shape=())
x = tf.case([(tf.equal(train_phase, 1), lambda: train_dataset),
             (tf.equal(train_phase, 2), lambda: validate_dataset),
             (tf.equal(train_phase,3), lambda: test_dataset)],
         default=lambda: train_dataset)
# ... construct model_op using tensor x as input ...
# ... call the initializer ...

# train for a bunch
for i in ...
     sess.run(main_op, feed_dict={train_phase: 1})
# validate
for i in ...
     sess.run(main_op, feed_dict={train_phase: 2})
# test
for i in ...
     sess.run(main_op, feed_dict={train_phase: 3})
</code></pre></div>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I am using a single tensor <code class="notranslate">x</code> to represent my input values for all three phases of my model training (training, validation, and testing). To alternate between them without duplicating entire graphs, I set the tensor to be conditional on the value of <code class="notranslate">train_phase</code> (again, train, validate, or test) by using <code class="notranslate">tf.case</code>.</p>
<p dir="auto">If the above steps are followed, my system will eventually (non-deterministically) crash during the training/validating/testing of the model. This is independent of the dataset used (I am using well-scrubbed versions of NYUv2, CIFAR-10, KITTI, etc.) and does not occur in particular records. The error is always long the lines of</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;, send_device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_69_add_1&quot;, tensor_type=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;]()]]
 Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;, send_device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_69_add_1&quot;, tensor_type=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;]()]]"><pre class="notranslate"><code class="notranslate">DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
 Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
</code></pre></div>
<p dir="auto">It can occur during any phase (that is, on any branch of the <code class="notranslate">tf.case</code>). It occurs with multiple versions of CUDA 8 and TF (at least since TFRecordDataset came out). It occurs with single or multiple GPUS. It occurs with different Linux kernel versions (multiple variants of 4.10, 4.11, 4.12, including the 4.9 TLS branch).</p>
<p dir="auto">However, it does <strong>NOT</strong> occur if I don't use <code class="notranslate">tf.case</code> (that is, I use a single TFRecordDataset tensor as the input, i.e., only do training and skip validate/test).</p>
<p dir="auto">I would assume this is a complication with the CUDA drivers, as I occasionally (though not always) get kernel panics at the same time. I will also occasionally (though not always) get errors like <code class="notranslate">malloc(): smallbin double linked list corrupted</code>.</p>
<p dir="auto">I should also add that I've already tested for hardware issues and seem to have ruled all of them out (memory failures, HD/SSD cables, motherboard, PSU spikes, etc.).</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">The full trace/output is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Caught unexpected exception during training: Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;, send_device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_69_add_1&quot;, tensor_type=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;]()]]

Caused by op 'input_pipeline/IteratorGetNext_1', defined at:
  File &quot;train.py&quot;, line 266, in &lt;module&gt;
    datasets = hem.get_datasets(args)
  File &quot;/mnt/research/projects/autoencoders/hem/util/data.py&quot;, line 64, in get_datasets
    x = iterator.get_next()
  File &quot;/usr/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py&quot;, line 311, in get_next
    name=name))
  File &quot;/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py&quot;, line 698, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File &quot;/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py&quot;, line 787, in _apply_op_helper
    op_def=op_def)
  File &quot;/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py&quot;, line 3046, in create_op
    op_def=op_def)
  File &quot;/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py&quot;, line 1604, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;, send_device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_69_add_1&quot;, tensor_type=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;]()]]
 Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;, send_device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_69_add_1&quot;, tensor_type=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:1&quot;]()]]"><pre class="notranslate"><code class="notranslate">Caught unexpected exception during training: Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Caused by op 'input_pipeline/IteratorGetNext_1', defined at:
  File "train.py", line 266, in &lt;module&gt;
    datasets = hem.get_datasets(args)
  File "/mnt/research/projects/autoencoders/hem/util/data.py", line 64, in get_datasets
    x = iterator.get_next()
  File "/usr/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 311, in get_next
    name=name))
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 698, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3046, in create_op
    op_def=op_def)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1604, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
 Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
	 [[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
	 [[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
</code></pre></div>
<p dir="auto">I try to write my own op and I have installed TensorFlow 0.12.0 with GPU support on Linux.</p>
<p dir="auto">This code fails:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#include &quot;tensorflow/core/platform/stream_executor.h&quot;"><pre class="notranslate"><code class="notranslate">#include "tensorflow/core/platform/stream_executor.h"
</code></pre></div>
<p dir="auto">With error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="fatal error: tensorflow/stream_executor/lib/status.h: No such file or directory"><pre class="notranslate"><code class="notranslate">fatal error: tensorflow/stream_executor/lib/status.h: No such file or directory
</code></pre></div>
<p dir="auto">That files does not exists.<br>
Some more include files seem to be missing. When grepping for <code class="notranslate">DeviceMemory</code> in the include path, the only file it finds is <code class="notranslate">include/tensorflow/core/util/stream_executor_util.h</code>.</p>