<p dir="auto">Hi,</p>
<p dir="auto">I'm trying to run this model on a system with GeForce GTX 960M:<br>
python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py<br>
and get the following output:</p>
<p dir="auto">I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally<br>
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 4<br>
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties:<br>
name: GeForce GTX 960M<br>
major: 5 minor: 0 memoryClockRate (GHz) 1.176<br>
pciBusID 0000:01:00.0<br>
Total memory: 2.00GiB<br>
Free memory: 1.69GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0:   Y<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0)<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 1.50GiB bytes.<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:52] GPU 0 memory begins at 0x501a40000 extends to 0x5616a0000<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00GiB<br>
I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 4<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity/_79 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_344_Identity", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Variable/initial_value/_81 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1185_Variable/initial_value", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv1/biases/Assign/_67 = _Send<a href="conv1/biases/Assign/_66">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_304_conv1/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_2/_63 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_293_Identity_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_3/_55 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_275_Identity_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv2/biases/Assign/_51 = _Send<a href="conv2/biases/Assign/_50">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_273_conv2/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_4/_47 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_262_Identity_4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_5/_39 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_233_Identity_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_6/_31 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_220_Identity_6", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local3/biases/Assign/_35 = _Send<a href="local3/biases/Assign/_34">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_231_local3/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_7/_23 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_191_Identity_7", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local4/biases/Assign/_19 = _Send<a href="local4/biases/Assign/_18">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_189_local4/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_8/_15 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_178_Identity_8", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_9/_7 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_160_Identity_9", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: softmax_linear/biases/Assign/_3 = _Send<a href="softmax_linear/biases/Assign/_2">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_softmax_linear/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Identity_1/_71 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_306_Identity_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv1/weights/Assign/_75 = _Send<a href="conv1/weights/Assign/_74">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_342_conv1/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: softmax_linear/weights/Assign/_11 = _Send<a href="softmax_linear/weights/Assign/_10">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_176_softmax_linear/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv1/biases/_64 = _Send<a href="conv1/biases">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_303_conv1/biases", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv2/biases/_48 = _Send<a href="conv2/biases">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_272_conv2/biases", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local3/biases/_32 = _Send<a href="local3/biases">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_230_local3/biases", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local4/biases/_16 = _Send<a href="local4/biases">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_188_local4/biases", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: softmax_linear/biases/_0 = _Send<a href="softmax_linear/biases">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_157_softmax_linear/biases", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: init/NoOp_1/_88 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1192_init/NoOp_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Variable/Assign/_84 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1186_Variable/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv1/weights/Assign/_76 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_342_conv1/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv1/biases/Assign/_68 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_304_conv1/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv2/weights/Assign/_60 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_291_conv2/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv2/biases/Assign/_52 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_273_conv2/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local3/weights/Assign/_44 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_260_local3/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local3/biases/Assign/_36 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_231_local3/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local4/weights/Assign/_28 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_218_local4/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local4/biases/Assign/_20 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_189_local4/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: softmax_linear/weights/Assign/_12 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_176_softmax_linear/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: softmax_linear/biases/Assign/_4 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_softmax_linear/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: conv2/weights/Assign/_59 = _Send<a href="conv2/weights/Assign/_58">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_291_conv2/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local4/weights/Assign/_27 = _Send<a href="local4/weights/Assign/_26">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_218_local4/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: local3/weights/Assign/_43 = _Send<a href="local3/weights/Assign/_42">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_260_local3/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
[[Node: Variable/initial_value/_80 = _HostSend<a href="Variable/initial_value">T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1185_Variable/initial_value", _device="/job:localhost/replica:0/task:0/gpu:0"</a>]]<br>
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.<br>
Traceback (most recent call last):<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 138, in <br>
tf.app.run()<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 30, in run<br>
sys.exit(main(sys.argv))<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 134, in main<br>
train()<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 94, in train<br>
sess.run(init)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 368, in run<br>
results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 444, in _do_run<br>
e.code)<br>
tensorflow.python.framework.errors.FailedPreconditionError: Attempting to use uninitialized value conv1/weights<br>
[[Node: conv1/weights/_72 = _Send<a href="conv1/weights">T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]</p>
<p dir="auto">Can somebody tell me what's wrong?</p>
<p dir="auto">The mnist model: tensorflow/models/image/mnist/convolutional.py<br>
seems to work well:</p>
<p dir="auto">I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally<br>
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 4<br>
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties:<br>
name: GeForce GTX 960M<br>
major: 5 minor: 0 memoryClockRate (GHz) 1.176<br>
pciBusID 0000:01:00.0<br>
Total memory: 2.00GiB<br>
Free memory: 1.71GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0:   Y<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0)<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 1.51GiB bytes.<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:52] GPU 0 memory begins at 0x501a40000 extends to 0x562600000<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.0KiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.00MiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00GiB<br>
I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 4<br>
Extracting data/train-images-idx3-ubyte.gz<br>
Extracting data/train-labels-idx1-ubyte.gz<br>
Extracting data/t10k-images-idx3-ubyte.gz<br>
Extracting data/t10k-labels-idx1-ubyte.gz<br>
Initialized!<br>
Epoch 0.00<br>
Minibatch loss: 12.054, learning rate: 0.010000<br>
Minibatch error: 90.6%<br>
Validation error: 84.6%<br>
Epoch 0.12<br>
Minibatch loss: 3.297, learning rate: 0.010000<br>
Minibatch error: 6.2%<br>
Validation error: 7.5%</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04.4 x64</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:  source master and source r1.9</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.9.0rc0 and 1.9.0rc2</li>
<li><strong>Python version</strong>:  3.6.6</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.15.0</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: 5.4.0</li>
<li><strong>CUDA/cuDNN version</strong>: 9.2/7.1</li>
<li><strong>GPU model and memory</strong>: GTX1080Ti 11GB  X 2</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<p dir="auto">checkout to r1.9 or master and ./configure</p>
<p dir="auto">after configure cuda and cudnn</p>
<p dir="auto">and configure NCCL ( default 1.3 or 2.2, whatever I choose, same )</p>
<p dir="auto">the process hangs forever</p>