Cool, thanks again for your great post. Big fan of your articles.<p>&gt; The functional graph becomes more complicated<p>Indeed. I once tried to come up with a &quot;higher order function&quot; that takes in a feedforward network and computes a separate function that computes the backward pass (like Theano&#x27;s autodiff, but abstraction at the layer-level rather than ops).<p>Here&#x27;s a diagram for a simple forward MLP (left to right), with the backward pass network below (right to left). I found this hard to work with, because of the explosion in the size of the computational graph when you try to decouple optimization &#x2F; function. I notice something similar when trying to unroll a RNN visually across time.
<a href="http:&#x2F;&#x2F;imgur.com&#x2F;ATNwknh" rel="nofollow">http:&#x2F;&#x2F;imgur.com&#x2F;ATNwknh</a><p>Let me know if this is way off base from what you were talking about in your post.