Neural Networks Are Essentially Polynomial Regression (2018)
When I was an AI grad student back in the late 80's and early 90's logic was all the rage.  People published papers about things like non-monotonic logics, intentional vs extensional logics, higher-order logics, the frame problem, circumscription...  One thing that always bothered me about the field was that the whole premise behind using logic as the basis for AI seemed to me to be <i>obviously</i> flawed because neither the inputs to nor the outputs from my brain were logical propositions, they were images and sounds on the input side, and sounds and gestures on the output side.  Even my internal reality didn't seem to consist of logical propositions.  When I went to open the refrigerator to make a sandwich, I didn't feel like I needed to prove any theorems in order to do it.  And I think subsequent developments have vindicated my skepticism.<p>Well, I have a similar objection to neural networks.  Human learning is fundamentally qualitatively different from contemporary neural network learning.  It is simply not the case that I learn what a cat looks like by being shown a bunch of images of cats and having some meta-input that says "label:cat".  Instead, I see a cat -- and a whole bunch of other things, including the environment that the cat happens to be in -- and I hear some sounds, and I somehow manage to make sense of the whole melange and sort out what a cat is and what the sound of my mother's voice is when she says, "That's a cat!, and the fact that the "cat" part of the audio stream is a label, and that it stands for this furry purry thing moving around in front of me, and that by producing some outputs that cause <i>me</i> to produce the same sounds from my own body I get the reward of hearing mommy say, "That's right!"<p>What we call neural networks today are interesting and useful, but I don't think they are even remotely close to anything interesting that the human brain does, and so quibbling over whether or not they are "essentially polynomial regression" or not is, IMHO, missing the point.<p>[EDIT: I had closed with "missing the point rather badly" but subsequent comments have convinced me that this extra emphasis was unwarranted.]