Automatic Differentiation (2009)
Can someone further clarify the distinction between automatic and symbolic differentiation?<p>The author claims that using symbolic differentiation on his example (f(x)=sin(x+sin(x+...+sin(x+sin(x))...))) builds a &quot;huge expression that would take much more time to compute&quot; than automatic differentiation.  At the top of the article there&#x27;s also an update noting an &quot;explosion of great tools for automatic differentiation&quot; recently, which I assume is referring to machine&#x2F;deep learning frameworks like Theano, TensorFlow, etc.  However, from my understanding, building an entire computational graph to express the gradient via the chain rule (backpropagation) is exactly what these frameworks do, and by the author&#x27;s brief description that seems to fall under symbolic differentiation.  So which one are these frameworks doing -- symbolic or automatic differentiation?  And, concretely, how would they differ if they were instead doing the other one?<p>Edit: I&#x27;ve been doing some further reading trying to understand this, and it&#x27;s only confusing me further.  Wikipedia has this diagram [1] which supposedly illustrates the relationship between the two -- what the hell is going on here?  The only interpretation I can come up with from it is that automatic differentiation takes code (written in a computer programming language) as input and produces code to compute the gradient, whereas in symbolic differentiation, the input and output are mathematical expressions.  But this doesn&#x27;t seem like a meaningful distinction to me...both math and computer code are languages; functions written in programming languages <i>are</i> mathematical functions (the ones that can actually be correctly called &quot;functions&quot;, anyway).  Someone please help me out here; what am I missing?<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Automatic_differentiation#&#x2F;media&#x2F;File:AutomaticDifferentiationNutshell.png" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Automatic_differentiation#&#x2F;med...</a>