Answers.Body: <p>In the eye, the retinal ganglion cells have a <a href="https://en.wikipedia.org/wiki/Receptive_field" rel="nofollow noreferrer">receptive field</a> that is equivalent to some types of convolution filters, most of them edge detectors.</p>
  <p>The brain is a big unknown, nobody knows how it does to organize, memorize, create concepts, learns the language, ... . Thus, it is not possible to establish a parallelism.</p>
  <p>In particular, brain has a capacity of handle independence in scale <strong>and rotation</strong> that CNN's are not able to reproduce.</p>
  <p>As general remark about NN and brain: even when it is always said that &quot;neural network cells&quot; are &quot;inspired&quot; in biological neurons, there are critical differences that made this similitude only a &quot;inspiration&quot;. Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison. The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the learned parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T11:16:33.367Z
Answers.Text: Your first and third paragraphs are interesting, particularly the third because some (many?) ANNs which have CNN sub-parts use them as input layers and these then feed into non-CNNs such as LSTMs.
Answers.UserDisplayName: Single Malt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-27T12:39:43.853Z
Answers.Text: Are you sure about brain (or at least our visual system) being invariant to rotation? I don't think that's the case (at least generally), as there are experiments that show we cannot properly recognise faces and texts if turned upside down. In that sense, CNNs also require data augmentation, which is basically training the too on rotated versions of the input data, and we can also learn to read upside down if trained to do so.
Answers.UserDisplayName: jjmontes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-27T12:42:49.353Z
Answers.Text: I'm not convinced about the fact that the brain does not have external aid or guidance for learning: we have hormones and other chemical mechanisms, and I wouldn't rule those out as candidate participants in our learning process?
Answers.UserDisplayName: jjmontes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T10:28:15.280Z
Answers.LastActivityDate: 2020-12-03T10:33:54.680Z
Answers.LastEditDate: 2020-12-03T10:33:54.680Z
Answers.UserDisplayName: pasaba por aqui
Answers.Body: <p>Yes, CNNs are inspired by the human brain [<a href="https://arxiv.org/pdf/1404.7828.pdf" rel="nofollow noreferrer">1</a>, <a href="https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf" rel="nofollow noreferrer">2</a>, <a href="https://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">3</a>]. More specifically, their operations, the convolution and pooling, are inspired by the human brain. However, note that, nowadays, CNNs are mainly trained with gradient descent (GD) and back-propagation (BP), which seems <strong>not</strong> to be a biologically plausible way of learning, but, given the success of GD and BP, there have been attempts to connect GB and BP with the way humans learn [<a href="https://brainscan.uwo.ca/research/cores/computational_core/uploads/11May2020-Lillicrap_NatNeuroRev_2020.pdf" rel="nofollow noreferrer">4</a>].</p>
  <p>The <strong>neocognitron</strong>, the first convolutional neural network [<a href="https://arxiv.org/pdf/1404.7828.pdf" rel="nofollow noreferrer">1</a>], proposed by <strong>Kunihiko Fukushima</strong> in 1979-1980, and described in the paper <a href="https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf" rel="nofollow noreferrer">Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position</a>, already uses convolutional and pooling (specifically, averaging pooling) layers [<a href="https://arxiv.org/pdf/1404.7828.pdf" rel="nofollow noreferrer">1</a>]. The neocognitron was inspired by the work of Hubel and Wiesel described in the 1959 paper <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/" rel="nofollow noreferrer">Receptive fields of single neurones in the cat's striate cortex</a>.</p>
  <p>Here is an excerpt from the 1980 Fukushima's paper.</p>
  <blockquote>
  <p>The mechanism of pattern recognition in the brain is little known, and it seems to be almost impossible to reveal it only by conventional physiological experiments. So, we take a slightly different approach to this problem. If we could make a neural network model which has the same capability for pattern recognition as a human being, it would give us a powerful clue to the understanding of the neural mechanism in the brain. In this paper, we discuss how to synthesize a neural network model in order to endow it an ability of pattern recognition like a human being.</p>
  <p>Several models were proposed with this intention (Rosenblatt, 1962; Kabrisky, 1966; Giebel, 1971; Fukushima, 1975). The response of most of these models, however, was severely affected by the shift in position and/or by the distortion in shape of the input patterns. Hence, their ability for pattern recognition
  was not so high.</p>
  <p>In this paper, we propose an improved neural network model. The structure of this network has been suggested by that of the visual nervous system of the vertebrate. This network is self-organized by &quot;learning without a teacher&quot;, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their position nor by small distortion of their shapes. This network is given a nickname &quot;neocognitron&quot;, because it is a further extention of the &quot;cognitron&quot;, which also is a self-organizing multilayered neural network model proposed by the author before (Fukushima, 1975)</p>
  </blockquote>
  <p>However, Fukushima did not train the neocognitron with gradient descent (and back-propagation) but with local learning rules (which are more biologically plausible), and that's probably why he doesn't get more credit, as I think he should.</p>
  <p>You should read at least Fukushima's paper for more details, which I will not replicate here.</p>
  <p><a href="https://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">Section 9.4 of the Deep Learning book</a> also contains details about how CNNs are inspired by neuroscience findings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T10:40:53.537Z
Answers.LastActivityDate: 2020-12-04T13:17:52.957Z
Answers.LastEditDate: 2020-12-04T13:17:52.957Z
Answers.UserDisplayName: nbro
Body: <p>The <a href="https://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">Deep Learning</a> book by Goodfellow et al. states</p>
  <blockquote>
  <p>Convolutional networks stand out as an example of neuroscientiﬁc principles inﬂuencing deep learning.</p>
  </blockquote>
  <p>Are convolutional neural networks (CNNs) really inspired by the human brain?</p>
  <p>If so, how? In particular, what structures within the brain do CNN-like neuron groupings occur?</p>
  
Tags: <convolutional-neural-networks><brain><neuroscience><neocognitron>
Title: Are convolutional neural networks inspired by the human brain?
UserDisplayName: Single Malt
ThreadText: # Are convolutional neural networks inspired by the human brain?
  
  (Asked by: username_0 on 2020-11-30)
  
  The [Deep Learning](https://www.deeplearningbook.org/contents/convnets.html) book by Goodfellow et al. states
  
  > Convolutional networks stand out as an example of neuroscientiﬁc principles inﬂuencing deep learning.
  
  Are convolutional neural networks (CNNs) really inspired by the human brain?
  
  If so, how? In particular, what structures within the brain do CNN-like neuron groupings occur?
  
  **Tags:** convolutional-neural-networks, brain, neuroscience, neocognitron
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-12-03. Score: 5 | Accepted answer
  
  Yes, CNNs are inspired by the human brain [[1](https://arxiv.org/pdf/1404.7828.pdf), [2](https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf), [3](https://www.deeplearningbook.org/contents/convnets.html)]. More specifically, their operations, the convolution and pooling, are inspired by the human brain. However, note that, nowadays, CNNs are mainly trained with gradient descent (GD) and back-propagation (BP), which seems **not** to be a biologically plausible way of learning, but, given the success of GD and BP, there have been attempts to connect GB and BP with the way humans learn [[4](https://brainscan.uwo.ca/research/cores/computational_core/uploads/11May2020-Lillicrap_NatNeuroRev_2020.pdf)].
  
  The **neocognitron**, the first convolutional neural network [[1](https://arxiv.org/pdf/1404.7828.pdf)], proposed by **Kunihiko Fukushima** in 1979-1980, and described in the paper [Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position](https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf), already uses convolutional and pooling (specifically, averaging pooling) layers [[1](https://arxiv.org/pdf/1404.7828.pdf)]. The neocognitron was inspired by the work of Hubel and Wiesel described in the 1959 paper [Receptive fields of username_0 neurones in the cat's striate cortex](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/).
  
  Here is an excerpt from the 1980 Fukushima's paper.
  
  > The mechanism of pattern recognition in the brain is little known, and it seems to be almost impossible to reveal it only by conventional physiological experiments. So, we take a slightly different approach to this problem. If we could make a neural network model which has the same capability for pattern recognition as a human being, it would give us a powerful clue to the understanding of the neural mechanism in the brain. In this paper, we discuss how to synthesize a neural network model in order to endow it an ability of pattern recognition like a human being.
  >
  > Several models were proposed with this intention (Rosenblatt, 1962; Kabrisky, 1966; Giebel, 1971; Fukushima, 1975). The response of most of these models, however, was severely affected by the shift in position and/or by the distortion in shape of the input patterns. Hence, their ability for pattern recognition
  > was not so high.
  >
  > In this paper, we propose an improved neural network model. The structure of this network has been suggested by that of the visual nervous system of the vertebrate. This network is self-organized by "learning without a teacher", and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their position nor by small distortion of their shapes. This network is given a nickname "neocognitron", because it is a further extention of the "cognitron", which also is a self-organizing multilayered neural network model proposed by the author before (Fukushima, 1975)
  
  However, Fukushima did not train the neocognitron with gradient descent (and back-propagation) but with local learning rules (which are more biologically plausible), and that's probably why he doesn't get more credit, as I think he should.
  
  You should read at least Fukushima's paper for more details, which I will not replicate here.
  
  [Section 9.4 of the Deep Learning book](https://www.deeplearningbook.org/contents/convnets.html) also contains details about how CNNs are inspired by neuroscience findings.
  
  
  ---
  
  ### Answer by username_1 on 2020-12-03. Score: 1
  
  In the eye, the retinal ganglion cells have a [receptive field](https://en.wikipedia.org/wiki/Receptive_field) that is equivalent to some types of convolution filters, most of them edge detectors.
  
  The brain is a big unknown, nobody knows how it does to organize, memorize, create concepts, learns the language, ... . Thus, it is not possible to establish a parallelism.
  
  In particular, brain has a capacity of handle independence in scale **and rotation** that CNN's are not able to reproduce.
  
  As general remark about NN and brain: even when it is always said that "neural network cells" are "inspired" in biological neurons, there are critical differences that made this similitude only a "inspiration". Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison. The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the learned parameters.
  
  > **Comments:**
  > **username_0**: Your first and third paragraphs are interesting, particularly the third because some (many?) ANNs which have CNN sub-parts use them as input layers and these then feed into non-CNNs such as LSTMs.  
  > **username_2**: Are you sure about brain (or at least our visual system) being invariant to rotation? I don't think that's the case (at least generally), as there are experiments that show we cannot properly recognise faces and texts if turned upside down. In that sense, CNNs also require data augmentation, which is basically training the too on rotated versions of the input data, and we can also learn to read upside down if trained to do so.  
  > **username_2**: I'm not convinced about the fact that the brain does not have external aid or guidance for learning: we have hormones and other chemical mechanisms, and I wouldn't rule those out as candidate participants in our learning process?  
  
  ---
  
  