<p dir="auto">ubuntu@slave1:/media/slave1temp/tensorflow_full$ python tensorflow/tensorflow/models/image/mnist/convolutional.py<br>
Extracting data/train-images-idx3-ubyte.gz<br>
Extracting data/train-labels-idx1-ubyte.gz<br>
Extracting data/t10k-images-idx3-ubyte.gz<br>
Extracting data/t10k-labels-idx1-ubyte.gz<br>
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 8<br>
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:888] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:88] Found device 0 with properties:<br>
name: GeForce GTX 750 Ti<br>
major: 5 minor: 0 memoryClockRate (GHz) 1.189<br>
pciBusID 0000:01:00.0<br>
Total memory: 2.00GiB<br>
Free memory: 1.96GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:112] DMA: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_init.cc:122] 0:   Y<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:643] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0)<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:47] Setting region size to 1896685568<br>
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 8<br>
Initialized!<br>
Epoch 0.00<br>
Minibatch loss: 12.054, learning rate: 0.010000<br>
Minibatch error: 90.6%<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 256 (256B) Pool: chunks: 64 free: 35 cumulative malloc: 93 cumulative freed: 64<br>
Number of chunks: 64, in_use chunks: 29<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 2048 (2.0KiB) Pool: chunks: 8 free: 4 cumulative malloc: 7 cumulative freed: 3<br>
Number of chunks: 8, in_use chunks: 4<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 4096 (4.0KiB) Pool: chunks: 16 free: 13 cumulative malloc: 18 cumulative freed: 15<br>
Number of chunks: 16, in_use chunks: 3<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 32768 (32.0KiB) Pool: chunks: 8 free: 5 cumulative malloc: 9 cumulative freed: 6<br>
Number of chunks: 8, in_use chunks: 3<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 139264 (136.0KiB) Pool: chunks: 15 free: 15 cumulative malloc: 16 cumulative freed: 16<br>
Number of chunks: 15, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 212992 (208.0KiB) Pool: chunks: 12 free: 9 cumulative malloc: 14 cumulative freed: 11<br>
Number of chunks: 12, in_use chunks: 3<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 278528 (272.0KiB) Pool: chunks: 1 free: 1 cumulative malloc: 1 cumulative freed: 1<br>
Number of chunks: 1, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 851968 (832.0KiB) Pool: chunks: 4 free: 4 cumulative malloc: 4 cumulative freed: 4<br>
Number of chunks: 4, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/17039362abcd0b2aeb05f148f99dc22e81faa256/hovercard" href="https://github.com/tensorflow/tensorflow/commit/17039362abcd0b2aeb05f148f99dc22e81faa256"><tt>1703936</tt></a> (1.62MiB) Pool: chunks: 6 free: 6 cumulative malloc: 6 cumulative freed: 6<br>
Number of chunks: 6, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 2883584 (2.75MiB) Pool: chunks: 1 free: 1 cumulative malloc: 1 cumulative freed: 1<br>
Number of chunks: 1, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/3407872626dc608f3d82ddbcf03f5c610646c1c1/hovercard" href="https://github.com/tensorflow/tensorflow/commit/3407872626dc608f3d82ddbcf03f5c610646c1c1"><tt>3407872</tt></a> (3.25MiB) Pool: chunks: 8 free: 8 cumulative malloc: 9 cumulative freed: 9<br>
Number of chunks: 8, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 6815744 (6.50MiB) Pool: chunks: 12 free: 9 cumulative malloc: 18 cumulative freed: 15<br>
Number of chunks: 12, in_use chunks: 3<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 15728640 (15.00MiB) Pool: chunks: 1 free: 0 cumulative malloc: 1 cumulative freed: 0<br>
Number of chunks: 1, in_use chunks: 1<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 20971520 (20.00MiB) Pool: chunks: 1 free: 1 cumulative malloc: 1 cumulative freed: 1<br>
Number of chunks: 1, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 125829120 (120.00MiB) Pool: chunks: 1 free: 0 cumulative malloc: 1 cumulative freed: 0<br>
Number of chunks: 1, in_use chunks: 1<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 251658240 (240.00MiB) Pool: chunks: 0 free: 0 cumulative malloc: 0 cumulative freed: 0<br>
Number of chunks: 0, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:339] Chunk size: 503316480 (480.00MiB) Pool: chunks: 3 free: 3 cumulative malloc: 3 cumulative freed: 3<br>
Number of chunks: 3, in_use chunks: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:345] Aggregate Region Memory: 1896685568 (1.77GiB)<br>
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:347] Aggregate Chunk Memory: 1803329536 (1.68GiB)<br>
W tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:89] Out of GPU memory, see memory state dump above<br>
W tensorflow/core/kernels/conv_ops.cc:162] Resource exhausted: OOM when allocating tensor with shapedim { size: 5000 } dim { size: 14 } dim { size: 14 } dim { size: 64 }<br>
W tensorflow/core/common_runtime/executor.cc:1027] 0x57b6090 Compute status: Resource exhausted: OOM when allocating tensor with shapedim { size: 5000 } dim { size: 14 } dim { size: 14 } dim { size: 64 }<br>
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](MaxPool_2, Variable_2)]]<br>
W tensorflow/core/common_runtime/executor.cc:1027] 0x50bc640 Compute status: Resource exhausted: OOM when allocating tensor with shapedim { size: 5000 } dim { size: 14 } dim { size: 14 } dim { size: 64 }<br>
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](MaxPool_2, Variable_2)]]<br>
[[Node: Softmax_1/_45 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_735_Softmax_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
Traceback (most recent call last):<br>
File "tensorflow/tensorflow/models/image/mnist/convolutional.py", line 270, in <br>
tf.app.run()<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 11, in run<br>
sys.exit(main(sys.argv))<br>
File "tensorflow/tensorflow/models/image/mnist/convolutional.py", line 258, in main<br>
validation_prediction.eval(), validation_labels)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 405, in eval<br>
return _eval_using_default_session(self, feed_dict, self.graph, session)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2728, in _eval_using_default_session<br>
return session.run(tensors, feed_dict)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 345, in run<br>
results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 419, in _do_run<br>
e.code)<br>
tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shapedim { size: 5000 } dim { size: 14 } dim { size: 14 } dim { size: 64 }<br>
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](MaxPool_2, Variable_2)]]<br>
[[Node: Softmax_1/_45 = _Recv<a href="">client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_735_Softmax_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"</a>]]<br>
Caused by op u'Conv2D_3', defined at:<br>
File "tensorflow/tensorflow/models/image/mnist/convolutional.py", line 270, in <br>
tf.app.run()<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 11, in run<br>
sys.exit(main(sys.argv))<br>
File "tensorflow/tensorflow/models/image/mnist/convolutional.py", line 229, in main<br>
validation_prediction = tf.nn.softmax(model(validation_data_node))<br>
File "tensorflow/tensorflow/models/image/mnist/convolutional.py", line 179, in model<br>
padding='SAME')<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 207, in conv2d<br>
use_cudnn_on_gpu=use_cudnn_on_gpu, name=name)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 633, in apply_op<br>
op_def=op_def)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1710, in create_op<br>
original_op=self._default_original_op, op_def=op_def)<br>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 988, in <strong>init</strong><br>
self._traceback = _extract_stack()</p>
<p dir="auto">ubuntu@slave1:/media/slave1temp/tensorflow_full$</p>
<p dir="auto">Using tensorflow nightly.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
with tf.name_scope('scope'):
    x = tf.Variable(42, dtype=tf.float32)
    ema = tf.train.ExponentialMovingAverage(decay=0.9)
    apply_op = ema.apply([x])
    average = ema.average(x)
    print average.name   # 'scope/scope/Variable/ExponentialMovingAverage:0'
    print ema.average_name(x)  # 'scope/Variable/ExponentialMovingAverage'"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">name_scope</span>(<span class="pl-s">'scope'</span>):
    <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-c1">42</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>)
    <span class="pl-s1">ema</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">ExponentialMovingAverage</span>(<span class="pl-s1">decay</span><span class="pl-c1">=</span><span class="pl-c1">0.9</span>)
    <span class="pl-s1">apply_op</span> <span class="pl-c1">=</span> <span class="pl-s1">ema</span>.<span class="pl-en">apply</span>([<span class="pl-s1">x</span>])
    <span class="pl-s1">average</span> <span class="pl-c1">=</span> <span class="pl-s1">ema</span>.<span class="pl-en">average</span>(<span class="pl-s1">x</span>)
    <span class="pl-k">print</span> <span class="pl-s1">average</span>.<span class="pl-s1">name</span>   <span class="pl-c"># 'scope/scope/Variable/ExponentialMovingAverage:0'</span>
    <span class="pl-k">print</span> <span class="pl-s1">ema</span>.<span class="pl-en">average_name</span>(<span class="pl-s1">x</span>)  <span class="pl-c"># 'scope/Variable/ExponentialMovingAverage'</span></pre></div>