Answers.Body: <p>In my experience researchers typically base their architectures on previously identified successful architectures and principles. That is to say published methods that have been successful in practice on similar tasks to the current one. This can be followed back to very early networks like the Perceptron which took a lot of inspiration from existing successful mathematical techniques.</p>
  <p>This is not to say that researchers do not draw from neuroscience or from broader knowledge to influence their decisions. I believe most research is guided by more abstract intuitions derived from broader experience. Some researchers are very strongly influenced by neuroscience and I understand that some try to replicate behaviours seen in the brain. The similar structure and functions of the brain will surely make knowledge of it useful. Some major advances may have been motivated by direct analogy to the brain, although I am not aware if that is the case. But, on balance, having zero knowledge of neuroscience is not an impediment to undertaking successful research into neural architectures. Having zero knowledge of existing successful techniques is a far greater impediment.</p>
  <p>Having a good level of mathematical understanding is also very useful. I will take perhaps a slight risk of going beyond my knowledge and suggest that university-level knowledge of mathematics is also more beneficial than knowledge of neuroscience. I think that is true for research so far, maybe it will change in the future. All these comments are based only on my personal experience and not on any formal research. Also, note my bias: I have minimal knowledge of neuroscience myself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-04T14:55:13.620Z
Answers.Text: For an example of a neural network that is based on a previously published neural network that was successful, you can simply look at all [variants of the LSTM that came after the original one in 1997](https://ai.stackexchange.com/a/14286/2444). For example, the LSTM with the peephole connections or even GRU.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-02T13:03:22.083Z
Answers.LastActivityDate: 2021-12-23T10:20:49.057Z
Answers.LastEditDate: 2021-12-23T10:20:49.057Z
Answers.UserDisplayName: user7834
Body: <p>I am not a researcher, but I am curious to know what considerations are relevant to take into account during research for the invention of a new neural network model, and what relevant knowledge researchers typically possess in the area.</p>
  <p>And an accompanying question: is a background in neuroscience relevant to such an investigation?</p>
  
Tags: <neural-networks><deep-learning><research>
Title: In the field of Deep Learning research, what considerations do researchers take into account when inventing new neural network models?
UserDisplayName: Andres Sarmiento
ThreadText: # In the field of Deep Learning research, what considerations do researchers take into account when inventing new neural network models?
  
  (Asked by: username_0 on 2021-06-02)
  
  I am not a researcher, but I am curious to know what considerations are relevant to take into account during research for the invention of a new neural network model, and what relevant knowledge researchers typically possess in the area.
  
  And an accompanying question: is a background in neuroscience relevant to such an investigation?
  
  **Tags:** neural-networks, deep-learning, research
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-02. Score: 4
  
  In my experience researchers typically base their architectures on previously identified successful architectures and principles. That is to say published methods that have been successful in practice on similar tasks to the current one. This can be followed back to very early networks like the Perceptron which took a lot of inspiration from existing successful mathematical techniques.
  
  This is not to say that researchers do not draw from neuroscience or from broader knowledge to influence their decisions. I believe most research is guided by more abstract intuitions derived from broader experience. Some researchers are very strongly influenced by neuroscience and I understand that some try to replicate behaviours seen in the brain. The similar structure and functions of the brain will surely make knowledge of it useful. Some major advances may have been motivated by direct analogy to the brain, although I am not aware if that is the case. But, on balance, having zero knowledge of neuroscience is not an impediment to undertaking successful research into neural architectures. Having zero knowledge of existing successful techniques is a far greater impediment.
  
  Having a good level of mathematical understanding is also very useful. I will take perhaps a slight risk of going beyond my knowledge and suggest that university-level knowledge of mathematics is also more beneficial than knowledge of neuroscience. I think that is true for research so far, maybe it will change in the future. All these comments are based only on my personal experience and not on any formal research. Also, note my bias: I have minimal knowledge of neuroscience myself.
  
  > **Comments:**
  > **username_2**: For an example of a neural network that is based on a previously published neural network that was successful, you can simply look at all [variants of the LSTM that came after the original one in 1997](https://ai.stackexchange.com/a/14286/2444). For example, the LSTM with the peephole connections or even GRU.  
  
  ---
  
  