<p dir="auto">The example</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
x = tf.Variable(1.)
A = tf.Variable(tf.ones((3,3))) 
cost = tf.trace(tf.foldl(tf.matmul,tf.stack([x*A for _ in range(3)])))
tf.gradients(tf.gradients(cost, A), x)  
# TypeError: Second-order gradient for while loops not supported."><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-c1">1.</span>)
<span class="pl-v">A</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-s1">tf</span>.<span class="pl-en">ones</span>((<span class="pl-c1">3</span>,<span class="pl-c1">3</span>))) 
<span class="pl-s1">cost</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">trace</span>(<span class="pl-s1">tf</span>.<span class="pl-en">foldl</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">matmul</span>,<span class="pl-s1">tf</span>.<span class="pl-en">stack</span>([<span class="pl-s1">x</span><span class="pl-c1">*</span><span class="pl-v">A</span> <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">3</span>)])))
<span class="pl-s1">tf</span>.<span class="pl-en">gradients</span>(<span class="pl-s1">tf</span>.<span class="pl-en">gradients</span>(<span class="pl-s1">cost</span>, <span class="pl-v">A</span>), <span class="pl-s1">x</span>)  
<span class="pl-c"># TypeError: Second-order gradient for while loops not supported.</span></pre></div>
<p dir="auto">illustrates that despite applying <code class="notranslate">tf.foldl</code> to a static list, the internal implementation via while loops leads to a type error. The problem disappears if the fold operation is carried out manually using a for loop. While implementing <code class="notranslate">foldl</code> using the while loop clearly makes the operation  more widely applicable, it seems problematic if syntactic sugar can lead to code that has qualitative differences from a naive implementation using a static loop. I cannot help but wonder whether <code class="notranslate">foldl</code> could be more efficient in the static case as well, although that is more of a conjecture.</p>
<p dir="auto">I think it would be nice if <code class="notranslate">foldl</code> (and other while loop derivatives) had a keyword that enabled or disabled the "dynamic mode" using while, or if, at the very least, the TypeError would occur at the <code class="notranslate">foldl</code>operation so that the error is easier to trace.</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:<br>
Custom code.</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
Linux Ubuntu 16.04 LTS</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:<br>
pip install.</li>
<li><strong>TensorFlow version (use command below)</strong>:<br>
v1.4.0-rc1-11-g130a514 1.4.0</li>
<li><strong>Python version</strong>:<br>
3.5.4</li>
<li><strong>Bazel version (if compiling from source)</strong>:<br>
Not applicable.</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:<br>
Not applicable.</li>
<li><strong>CUDA/cuDNN version</strong>:<br>
Did not use CUDA.</li>
<li><strong>GPU model and memory</strong>:<br>
Did not use GPU.</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
x = tf.Variable(1.)
A = tf.Variable(tf.ones((3,3))) 
cost = tf.trace(tf.foldl(tf.matmul,tf.stack([x*A for _ in range(3)])))
tf.gradients(tf.gradients(cost, A), x)  
# TypeError: Second-order gradient for while loops not supported."><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-c1">1.</span>)
<span class="pl-v">A</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-s1">tf</span>.<span class="pl-en">ones</span>((<span class="pl-c1">3</span>,<span class="pl-c1">3</span>))) 
<span class="pl-s1">cost</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">trace</span>(<span class="pl-s1">tf</span>.<span class="pl-en">foldl</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">matmul</span>,<span class="pl-s1">tf</span>.<span class="pl-en">stack</span>([<span class="pl-s1">x</span><span class="pl-c1">*</span><span class="pl-v">A</span> <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">3</span>)])))
<span class="pl-s1">tf</span>.<span class="pl-en">gradients</span>(<span class="pl-s1">tf</span>.<span class="pl-en">gradients</span>(<span class="pl-s1">cost</span>, <span class="pl-v">A</span>), <span class="pl-s1">x</span>)  
<span class="pl-c"># TypeError: Second-order gradient for while loops not supported.</span></pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: No</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 10</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Binary</li>
<li><strong>TensorFlow version (use command below)</strong>: unknown, command fails (this is the problem)</li>
<li><strong>Bazel version (if compiling from source)</strong>: N/A</li>
<li><strong>CUDA/cuDNN version</strong>: 8.0/5.1</li>
<li><strong>GPU model and memory</strong>: GeForce GTX 970, 4 GB</li>
<li><strong>Exact command to reproduce</strong>: import tensorflow as tf</li>
</ul>
<h3 dir="auto">Problem</h3>
<p dir="auto">This is something I've seen in a <a href="https://github.com/tensorflow/tensorflow/issues/5949" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/5949/hovercard">previous issue report</a> added as a comment, but no one responded to it (except to say that you should be using cuDNN 5.1 instead of 6.0, which I am doing). I cannot even import tensorflow because it leads to the string of errors below. Someone in the linked issue suggested that this was caused by a missing MSVCP140.DLL, but this file is present in both my System32 and SysWOW64 folders (I have seen both mentioned). I have also installed the suggested Visual C++ Redistributable Update 3 with no luck.</p>
<h3 dir="auto">Log</h3>
<p dir="auto">Traceback (most recent call last):<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper<br>
return importlib.import_module(mname)<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\importlib_<em>init</em>_.py", line 126, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
File "", line 986, in _gcd_import<br>
File "", line 969, in _find_and_load<br>
File "", line 958, in _find_and_load_unlocked<br>
File "", line 666, in _load_unlocked<br>
File "", line 577, in module_from_spec<br>
File "", line 906, in create_module<br>
File "", line 222, in _call_with_frames_removed<br>
ImportError: DLL load failed: The specified module could not be found.</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <br>
from tensorflow.python.pywrap_tensorflow_internal import *<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <br>
_pywrap_tensorflow_internal = swig_import_helper()<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper<br>
return importlib.import_module('<em>pywrap_tensorflow_internal')<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\importlib_<em>init</em></em>.py", line 126, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
ImportError: No module named '_pywrap_tensorflow_internal'</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "", line 1, in <br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow_<em>init</em>_.py", line 24, in <br>
from tensorflow.python import *<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python_<em>init</em>_.py", line 51, in <br>
from tensorflow.python import pywrap_tensorflow<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 52, in <br>
raise ImportError(msg)<br>
ImportError: Traceback (most recent call last):<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper<br>
return importlib.import_module(mname)<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\importlib_<em>init</em>_.py", line 126, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
File "", line 986, in _gcd_import<br>
File "", line 969, in _find_and_load<br>
File "", line 958, in _find_and_load_unlocked<br>
File "", line 666, in _load_unlocked<br>
File "", line 577, in module_from_spec<br>
File "", line 906, in create_module<br>
File "", line 222, in _call_with_frames_removed<br>
ImportError: DLL load failed: The specified module could not be found.</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <br>
from tensorflow.python.pywrap_tensorflow_internal import *<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <br>
_pywrap_tensorflow_internal = swig_import_helper()<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper<br>
return importlib.import_module('<em>pywrap_tensorflow_internal')<br>
File "C:\Users\Elijah\AppData\Local\Programs\Python\Python35\lib\importlib_<em>init</em></em>.py", line 126, in import_module<br>
return _bootstrap._gcd_import(name[level:], package, level)<br>
ImportError: No module named '_pywrap_tensorflow_internal'</p>