Answers.Body: <p>In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, <strong>electrical engineering</strong>, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits.</p>
  <p>There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1], and a 1956 paper by Rochester et al. [2].</p>
  <p>The latter paper uses terms such as 'circuits' and 'switching'.  One idea in the paper is explained in terms of a &quot;Eccles-Jordan Flip Flop circuit&quot; although there are no drawings.  Nathanial Rochester had designed the IBM 701k [3] and &quot;led the first effort to simulate a neural network&quot; [4]</p>
  <p>Brain structure was discussed in terms of 'neural circuits' as early as 1937 [5].</p>
  <p>I am not sure when the first electrical circuit diagram appeared in publication, but it makes sense that early neural network designers, would have thought of their implementation as such.</p>
  <p>References:</p>
  <ul>
  <li>[1] McCulloch, W. S., and Walter Pitts, W. (1943) A Logical Calculus of the Ideas Immanent in Nervous Activity. <a href="https://jontalle.web.engr.illinois.edu/uploads/498-NS.21/McCulloch-Pitts-1943-neural-networks-ocr.pdf" rel="nofollow noreferrer">https://jontalle.web.engr.illinois.edu/uploads/498-NS.21/McCulloch-Pitts-1943-neural-networks-ocr.pdf</a></li>
  <li>[2] Rochester, N., Holland, J., Haibt, L., &amp; Duda, W. (1956). Tests on a cell assembly theory of the action of the brain, using a large digital computer. IRE Transactions on Information Theory, Information Theory, IRE Transactions on, IRE Trans. Inf. Theory, 2(3), 80–93. https://doi-org/10.1109/TIT.1956.1056810</li>
  <li>[3] Brief History of Neural Networks
  <a href="https://www.linkedin.com/pulse/data-science-term-neural-networks-70-year-old-anchit-sharma/?trk=public_profile_article_view" rel="nofollow noreferrer">https://www.linkedin.com/pulse/data-science-term-neural-networks-70-year-old-anchit-sharma/?trk=public_profile_article_view</a></li>
  <li>[4] This Data Science term - Neural Networks is 70-Year-Old Technology!
  <a href="https://medium.com/analytics-vidhya/brief-history-of-neural-networks-44c2bf72eec" rel="nofollow noreferrer">https://medium.com/analytics-vidhya/brief-history-of-neural-networks-44c2bf72eec</a>:</li>
  <li>[5] Papez circuit, <a href="https://en.wikipedia.org/wiki/Papez_circuit" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Papez_circuit</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T15:58:11.360Z
Answers.LastActivityDate: 2021-04-30T15:58:11.360Z
Answers.UserDisplayName: Brian O'Donnell
Answers.Body: <p>Well, the goal of any paper is to allow the reader to understand what the author is trying to describe.</p>
  <p>A lot of people have a lot of experience looking at circuit diagrams and figuring out those circuits will do. For these people, a circuit diagram may be the clearest and easiest way for them to understand how a particular thing works. So, it makes sense that an author would include a circuit diagram, in order to make it easy for those people to understand the concepts.</p>
  <p>There are two particular reasons why a circuit diagram is especially likely to show up in a paper about neural networks:</p>
  <p>The first reason is that analog circuits are closely related to <a href="https://en.wikipedia.org/wiki/Ordinary_differential_equation" rel="nofollow noreferrer">ordinary differential equations</a>, and digital circuits are closely related to <a href="https://en.wikipedia.org/wiki/Sequential_logic" rel="nofollow noreferrer">sequential logic</a>. So, if you have a neural network or something that uses ordinary differential equations or sequential logic, then a circuit diagram might be a simple way to express how it works.</p>
  <p>The second reason is that a lot of researchers who are familiar with computers are also familiar with electronic circuits. This was especially true in the early days of computers, when people <em>had</em> to be familiar with electronics, math, or both in order to understand how to program a computer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T17:02:30.937Z
Answers.Text: Thanks for your answer. Can you give more detail or explanation or reference that circuit diagrams are closely related ODE system?
Answers.UserDisplayName: dawen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T17:27:30.037Z
Answers.Text: @dawen That's the material that would be covered in an "introduction to electronic circuits" course.
Answers.UserDisplayName: Sophie Swett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T15:44:34.023Z
Answers.LastActivityDate: 2021-04-30T15:44:34.023Z
Answers.UserDisplayName: Sophie Swett
Body: <p>I am familiar with the currently popular neural network in deep learning, which has weights and is trained by gradient descent.</p>
  <p>However, I found many papers that were popular in the 1980s and 1990s.
  These papers have titles like &quot;Neural networks to solve optimization problems&quot;.
  For example, Hopfield first use this name, and they used &quot;Neural network&quot; to solve linear programming problems [1].
  Later, Kennedy et.al used &quot;Neural network&quot; to solve nonlinear programming problems [2].</p>
  <p>I summarize the difference between the current popular neural network and the &quot;Neural networks&quot;:</p>
  <ol>
  <li>They do not have parameter weights and bias to train or to learn from data.</li>
  <li>They used a circuit diagram to present the model.</li>
  <li>The model can be simplified as an ODE system and has a Lyapunov function as objective.</li>
  </ol>
  <p>Please take a look at these two papers in the 1980s:</p>
  <ol>
  <li><p>[Neurons with graded response have computational properties like those of two state neurons][2] (J.J. Hopfield)</p>
  </li>
  <li><p><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1783" rel="nofollow noreferrer">Neural Networks for Non-linear Programming</a> (M.P Kennedy &amp; L.O Chua)</p>
  </li>
  </ol>
  <p>Reference:</p>
  <p>[1]: J. J. Hopfield, D. W. Tank, “neural” computation of decisions in optimization problems, Biological265 cybernetics 52 (3) (1985) 141–152.</p>
  <p>[2]: M. P. Kennedy, L. O. Chua, Neural networks for nonlinear programming, IEEE Transactions on Circuits and Systems 35 (5) (1988) 554–562.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-29T21:29:35.497Z
Comments.Text: You seem to be asking us why John Hopfield used a circuit diagram when he wrote his paper describing his neural networks. Given that we're not able to read Hopfield's mind, I don't think it's possible to answer this question (aside from giving the obvious answer, which is "because Hopfield thought that was a good way to describe his neural networks").
Comments.UserDisplayName: Sophie Swett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T10:56:02.060Z
Comments.Text: I agree with the previous comment. I don't really understand what your question really is here. Can you clarify that? Moreover, in the body, you only talk about Hopfield networks, while in the title you're talking about "some neural networks". Which ones, apart from the Hopfield network? In other words, if you're just referring to the Hopfield network, don't use "some", but be as much specific as possible.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T12:31:27.150Z
Comments.Text: 1. I don't agree with the user 'nbro' comment. I believe I have stated the question clearly. I am asking about a general type of neural network models(many) presented by circuit diagram,  and I referred to the Hopfield network(one) as a classic example. The Hopfield network(one) obviously belongs to the general type of neural network models(many) I was talking about. I don't see any confusion about this.
Comments.UserDisplayName: dawen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T12:38:29.317Z
Comments.Text: 2. I don't agree with the user ' Tanner Swett' comment. It is possible to answer the question. You don't need to go inside someone brain to understand his thought. There must be some reason or some benefit to using the circuit diagram, the reason may state somewhere I don't know. And this is why I am asking the question here, hope someone knows it can give an answer.
Comments.UserDisplayName: dawen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-14T07:09:09.703Z
Comments.Text: I've rolled back an edit that changed the question and invalidated the existing answers. In the future, please don't substantially change the question after answers have already been posted. Thanks!
Comments.UserDisplayName: Mithical
Tags: <neural-networks><papers><neurons><hopfield-network>
Title: Why some neural network models in the 1980s shown as circuit models
UserDisplayName: dawen
ThreadText: # Why some neural network models in the 1980s shown as circuit models
  
  (Asked by: username_0 on 2021-04-29)
  
  I am familiar with the currently popular neural network in deep learning, which has weights and is trained by gradient descent.
  
  However, I found many papers that were popular in the 1980s and 1990s.
  These papers have titles like "Neural networks to solve optimization problems".
  For example, Hopfield first use this name, and they used "Neural network" to solve linear programming problems [1].
  Later, Kennedy et.al used "Neural network" to solve nonlinear programming problems [2].
  
  I summarize the difference between the current popular neural network and the "Neural networks":
  
  1. They do not have parameter weights and bias to train or to learn from data.
  2. They used a circuit diagram to present the model.
  3. The model can be simplified as an ODE system and has a Lyapunov function as objective.
  
  Please take a look at these two papers in the 1980s:
  
  1. [Neurons with graded response have computational properties like those of two state neurons][2] (J.J. Hopfield)
  2. [Neural Networks for Non-linear Programming](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1783) (M.P Kennedy & L.O Chua)
  
  Reference:
  
  [1]: J. J. Hopfield, D. W. Tank, “neural” computation of decisions in optimization problems, Biological265 cybernetics 52 (3) (1985) 141–152.
  
  [2]: M. P. Kennedy, L. O. Chua, Neural networks for nonlinear programming, IEEE Transactions on Circuits and Systems 35 (5) (1988) 554–562.
  
  **Tags:** neural-networks, papers, neurons, hopfield-network
  > **Comments:**
  > **username_1**: You seem to be asking us why John Hopfield used a circuit diagram when he wrote his paper describing his neural networks. Given that we're not able to read Hopfield's mind, I don't think it's possible to answer this question (aside from giving the obvious answer, which is "because Hopfield thought that was a good way to describe his neural networks").  
  > **username_2**: I agree with the previous comment. I don't really understand what your question really is here. Can you clarify that? Moreover, in the body, you only talk about Hopfield networks, while in the title you're talking about "some neural networks". Which ones, apart from the Hopfield network? In other words, if you're just referring to the Hopfield network, don't use "some", but be as much specific as possible.  
  > **username_0**: 1. I don't agree with the user 'nbro' comment. I believe I have stated the question clearly. I am asking about a general type of neural network models(many) presented by circuit diagram,  and I referred to the Hopfield network(one) as a classic example. The Hopfield network(one) obviously belongs to the general type of neural network models(many) I was talking about. I don't see any confusion about this.  
  > **username_0**: 2. I don't agree with the user ' Tanner Swett' comment. It is possible to answer the question. You don't need to go inside someone brain to understand his thought. There must be some reason or some benefit to using the circuit diagram, the reason may state somewhere I don't know. And this is why I am asking the question here, hope someone knows it can give an answer.  
  > **username_3**: I've rolled back an edit that changed the question and invalidated the existing answers. In the future, please don't substantially change the question after answers have already been posted. Thanks!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2021-04-30. Score: 3
  
  In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, **electrical engineering**, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits.
  
  There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1], and a 1956 paper by Rochester et al. [2].
  
  The latter paper uses terms such as 'circuits' and 'switching'. One idea in the paper is explained in terms of a "Eccles-Jordan Flip Flop circuit" although there are no drawings. Nathanial Rochester had designed the IBM 701k [3] and "led the first effort to simulate a neural network" [4]
  
  Brain structure was discussed in terms of 'neural circuits' as early as 1937 [5].
  
  I am not sure when the first electrical circuit diagram appeared in publication, but it makes sense that early neural network designers, would have thought of their implementation as such.
  
  References:
  
  * [1] McCulloch, W. S., and Walter Pitts, W. (1943) A Logical Calculus of the Ideas Immanent in Nervous Activity. <https://jontalle.web.engr.illinois.edu/uploads/498-NS.21/McCulloch-Pitts-1943-neural-networks-ocr.pdf>
  * [2] Rochester, N., Holland, J., Haibt, L., & Duda, W. (1956). Tests on a cell assembly theory of the action of the brain, using a large digital computer. IRE Transactions on Information Theory, Information Theory, IRE Transactions on, IRE Trans. Inf. Theory, 2(3), 80–93. https://doi-org/10.1109/TIT.1956.1056810
  * [3] Brief History of Neural Networks
    <https://www.linkedin.com/pulse/data-science-term-neural-networks-70-year-old-anchit-sharma/?trk=public_profile_article_view>
  * [4] This Data Science term - Neural Networks is 70-Year-Old Technology!
    <https://medium.com/analytics-vidhya/brief-history-of-neural-networks-44c2bf72eec>:
  * [5] Papez circuit, <https://en.wikipedia.org/wiki/Papez_circuit>
  
  
  ---
  
  ### Answer by username_1 on 2021-04-30. Score: 2
  
  Well, the goal of any paper is to allow the reader to understand what the author is trying to describe.
  
  A lot of people have a lot of experience looking at circuit diagrams and figuring out those circuits will do. For these people, a circuit diagram may be the clearest and easiest way for them to understand how a particular thing works. So, it makes sense that an author would include a circuit diagram, in order to make it easy for those people to understand the concepts.
  
  There are two particular reasons why a circuit diagram is especially likely to show up in a paper about neural networks:
  
  The first reason is that analog circuits are closely related to [ordinary differential equations](https://en.wikipedia.org/wiki/Ordinary_differential_equation), and digital circuits are closely related to [sequential logic](https://en.wikipedia.org/wiki/Sequential_logic). So, if you have a neural network or something that uses ordinary differential equations or sequential logic, then a circuit diagram might be a simple way to express how it works.
  
  The second reason is that a lot of researchers who are familiar with computers are also familiar with electronic circuits. This was especially true in the early days of computers, when people *had* to be familiar with electronics, math, or both in order to understand how to program a computer.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. Can you give more detail or explanation or reference that circuit diagrams are closely related ODE system?  
  > **username_1**: @dawen That's the material that would be covered in an "introduction to electronic circuits" course.  
  
  ---
  
  