<p dir="auto">Tensorflow is failing like so - very odd since I have memory available and it sees that. This runs fine in CPU only.</p>
<p dir="auto">Ubuntu 16.04, Cuda 8.0, CUDNN 5.1 for 8.0, Nvidia 367.57 driver, tensorflow_gpu-0.12.0rc0-cp27-none-linux_x86_64.whl. The rest you can see in the log. I have also tried with CUDNN 5.0 with the same result. Cuda 7.5 works for me but is very slow.</p>
<p dir="auto"><strong>Log:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
Exceptions: 0
1403
1403
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 660
major: 3 minor: 0 memoryClockRate (GHz) 1.0975
pciBusID 0000:02:00.0
Total memory: 1.99GiB
Free memory: 1.43GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x2c47900
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: 
name: GeForce GTX 660
major: 3 minor: 0 memoryClockRate (GHz) 1.0975
pciBusID 0000:01:00.0
Total memory: 1.99GiB
Free memory: 1.41GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 0 and 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 1 and 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y N 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   N Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 660, pci bus id: 0000:02:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -&gt; (device: 1, name: GeForce GTX 660, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 198.83M (208486400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
Traceback (most recent call last):
  File &quot;build.py&quot;, line 84, in &lt;module&gt;
    model = tflearn.DNN(net)
  File &quot;/usr/local/lib/python2.7/dist-packages/tflearn/models/dnn.py&quot;, line 63, in __init__
    best_val_accuracy=best_val_accuracy)
  File &quot;/usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py&quot;, line 135, in __init__
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py&quot;, line 1000, in __init__
    self.build()
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py&quot;, line 1021, in build
    raise ValueError(&quot;No variables to save&quot;)
ValueError: No variables to save"><pre class="notranslate"><code class="notranslate">I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
Exceptions: 0
1403
1403
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 660
major: 3 minor: 0 memoryClockRate (GHz) 1.0975
pciBusID 0000:02:00.0
Total memory: 1.99GiB
Free memory: 1.43GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x2c47900
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: 
name: GeForce GTX 660
major: 3 minor: 0 memoryClockRate (GHz) 1.0975
pciBusID 0000:01:00.0
Total memory: 1.99GiB
Free memory: 1.41GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 0 and 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 1 and 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y N 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   N Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 660, pci bus id: 0000:02:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -&gt; (device: 1, name: GeForce GTX 660, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 198.83M (208486400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
Traceback (most recent call last):
  File "build.py", line 84, in &lt;module&gt;
    model = tflearn.DNN(net)
  File "/usr/local/lib/python2.7/dist-packages/tflearn/models/dnn.py", line 63, in __init__
    best_val_accuracy=best_val_accuracy)
  File "/usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py", line 135, in __init__
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1000, in __init__
    self.build()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1021, in build
    raise ValueError("No variables to save")
ValueError: No variables to save
</code></pre></div>
<p dir="auto"><strong>Code:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="net = tflearn.input_data(shape=[None, 5])
net = tflearn.fully_connected(net, 64)
net = tflearn.fully_connected(net, 64)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net)

# Training
model = tflearn.DNN(net)
model.fit(X, Y, n_epoch=10000, batch_size=64, show_metric=True)

for x in range(0,50):
    rand = randint(0,len(X))
    print(model.predict([X[rand]]), Y[rand])

model.save(&quot;model.tfl&quot;)"><pre class="notranslate"><code class="notranslate">net = tflearn.input_data(shape=[None, 5])
net = tflearn.fully_connected(net, 64)
net = tflearn.fully_connected(net, 64)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net)

# Training
model = tflearn.DNN(net)
model.fit(X, Y, n_epoch=10000, batch_size=64, show_metric=True)

for x in range(0,50):
    rand = randint(0,len(X))
    print(model.predict([X[rand]]), Y[rand])

model.save("model.tfl")
</code></pre></div>
<p dir="auto">Hi all,</p>
<p dir="auto">Recently, I read some papers about implementing Intel Xeon Phi on Tensorflow. Therefore, I want to make TensorFlow support FPGA board. But there is no information in Google and other forum/community.</p>
<p dir="auto">For my understanding, we should implement the FPGA supporting in a class of TensorFlow which distributes tasks to different devices (like GPU/CPU). Is possible that implementing the FPGA supporting in that class? Or is there any good suggestions to make the progress to support FPGA on TensorFlow?</p>
<p dir="auto">I am looking forward any help or advice!</p>
<p dir="auto">Thanks,<br>
Kevin</p>