<p dir="auto">I'm trying to implement a 3d fully convolutional network on my GPU. But for some reason I get a crash.</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu 14.04 LTS<br>
GPU: GeForce Titan X .</p>
<p dir="auto">Installed version of CUDA and cuDNN: 8.0 and 5<br>
(attach the output of `ls -l /path/to/cuda/lib/libcud*<br>
<a href="https://github.com/tensorflow/tensorflow/files/599236/cud.filelist.txt">cud.filelist.txt</a> )</p>
<p dir="auto">I installed tensorflow version 0.11.0rc2, and it also reproduce in docker installation (gcr.io/tensorflow/tensorflow:latest-gpu)</p>
<h3 dir="auto">Example code</h3>
<p dir="auto">The following code reproduce the problem:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
import tensorflow as tf

graph = tf.Graph()

with graph.as_default():
    tf_dataset = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))
    tf_label = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))

    layer1_weights = tf.Variable(tf.truncated_normal((2, 2, 2, 1, 1), stddev=0.1))
    layer1_bias = tf.Variable(tf.zeros(1))

    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
    logits = tf.nn.relu(conv+layer1_bias)

    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_label))
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    batchData = np.random.rand(1, 512, 512, 512, 1).astype(np.float32)
    batchLabels = (np.random.rand(1, 512, 512, 512, 1)&gt;0.5).astype(np.float32)
    feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
    _ = session.run((optimizer, ), feed_dict=feed_dict)"><pre class="notranslate"><code class="notranslate">import numpy as np
import tensorflow as tf

graph = tf.Graph()

with graph.as_default():
    tf_dataset = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))
    tf_label = tf.placeholder(tf.float32, shape=(1, 512, 512, 512, 1))

    layer1_weights = tf.Variable(tf.truncated_normal((2, 2, 2, 1, 1), stddev=0.1))
    layer1_bias = tf.Variable(tf.zeros(1))

    conv = tf.nn.conv3d(tf_dataset, layer1_weights, (1, 1, 1, 1, 1), padding='SAME')
    logits = tf.nn.relu(conv+layer1_bias)

    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_label))
    optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    batchData = np.random.rand(1, 512, 512, 512, 1).astype(np.float32)
    batchLabels = (np.random.rand(1, 512, 512, 512, 1)&gt;0.5).astype(np.float32)
    feed_dict = {tf_dataset : batchData, tf_label : batchLabels}
    _ = session.run((optimizer, ), feed_dict=feed_dict)
</code></pre></div>
<p dir="auto">with the following output:</p>
<blockquote>
<p dir="auto">I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:<br>
name: GeForce GTX TITAN X<br>
major: 5 minor: 2 memoryClockRate (GHz) 1.076<br>
pciBusID 0000:01:00.0<br>
Total memory: 11.92GiB<br>
Free memory: 11.68GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)<br>
F tensorflow/stream_executor/cuda/cuda_dnn.cc:2440] failed to enqueue convolution on stream: CUDNN_STATUS_NOT_SUPPORTED</p>
</blockquote>
<p dir="auto">when my graph has the digamma function</p>