Backpropagation is a leaky abstraction
I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff &quot;the hard way&quot; - so we&#x27;d understand what was going on later when we used Octave&#x27;s methods.<p>Something about this article as a whole, though, does raise a question that I&#x27;ve been wondering about, and I want to present it here for a bit of discussion (maybe it needs its own thread?):<p>Does anyone else here think that the current approach to neural networks has some fundamental flaws?<p>Now - I&#x27;m not an expert; call me an interested student right now, with only the barest of experience (beyond the ML Class, I also took Udacity&#x27;s CS373 course, and I am also currently enrolled in their Self-Driving Car Engineer nanodegree program).<p>I understand that what we currently have and know does work. What I mean by that is the basic idea of an artificial neural network using forward and back-prop, multiple layers, etc (and all the derivatives - RNN, CNN, deep learning, etc). I understand the need and reasoning behind using activation functions based around calculus and derivatives and the chain-rule, etc (though I admit I need further education in these items).<p>But something nags at me.<p>All of this, despite the fact that it works and works well (provided all your tuning and such is right, etc), just seems like it is over-complicated. Real neurons don&#x27;t use calculus and activation functions, nor back-propagation, etc in order to learn. All of those things in an ANN are just abstractions and models around what occurs in nature.<p>Maybe (probably?) I am wrong - but it seems like what nature does is simpler. Much less power is used, for instance, and the package is much more compact. I just have this feeling that in some manner we may have gone down a path that while it has produced a working representation, that representation is overly complex, and had we took another approach (whatever that might be?), our ANNs would look and work much differently - perhaps even more efficiently.<p>About the only alternatives I have heard about otherwise have been things like spike-train neural networks, and some of the other &quot;closer to nature&quot; simulation (of ion pumps and real synapses, etc). Still, even those, while seemingly closer, also have what appears to be too much complexity.<p>I&#x27;m probably just talking out of my nether regions as a general n00b to the field. I do wonder, though, if there might be another solution, seemingly out in &quot;left-field&quot; that might push things forward, if someone was willing to look and experiment. It is something I plan to look into myself, as I find time and such between lessons and other work for my current learning experience.