Answers.Body: <p>I will first address your main question &quot;Why did the development of neural networks stop between 50s and 80s?&quot;
  In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)!</p>
  <p>However, Minsky and Papert have proved in 1969 that a single-layer architecture is not enough to build a universal approximating machine (see e.g. Minsky, M. &amp; Papert, S. Perceptrons: An Introduction to Computational Geometry, vol. 165 (1969)). That led to the first disappointment in &quot;AI&quot;. Which lasted until several major breakthroughs in 1980s: the proof of universal approximating capabilities of multi-layer perceptron by Cybenko, the popularisation of backpropagation algorithm (Hinton and colleagues), etc.</p>
  <p>I agree with LeCun that using continuous activation functions have enabled the backpropagation algorithm at the time. It is only recently that we have learned to <a href="https://arxiv.org/pdf/1602.02830.pdf" rel="nofollow noreferrer">backpropagate in networks with binary activation functions</a> (2016!).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T01:01:43.887Z
Answers.Text: Thank you for your answer! LeCunn also says in the same lecture that we can approximate any function with an SVM, is it related to the universal approximating machine you mentioned ? And do you perhaps know in which article it was proved that we can approximate any function with an SVM please ? And that's crazy how it's until recently researchers found a method to train networks with binary activation functions. Like it was theoretically known that backpropagation works with discontinuous functions but we weren't able to do so in practice.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-14T13:43:02.177Z
Answers.Text: He probably refers to the Universal approximation theorem https://en.wikipedia.org/wiki/Universal_approximation_theorem. You may want to read e.g. this article http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-14T13:58:10.267Z
Answers.Text: Thank you for your clarification and for the article. (I tried to cite you penkovsky with the @ but it's not working I'm sorry)
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-14T13:59:32.367Z
Answers.Text: Happy to help. You may want to accept the answer if you find it satisfying.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-16T00:01:48.353Z
Answers.Text: Yes that's right, I always forget to do it. Thank you !
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T12:17:57.180Z
Answers.LastActivityDate: 2021-03-17T09:48:56.583Z
Answers.LastEditDate: 2021-03-17T09:48:56.583Z
Answers.UserDisplayName: penkovsky
Body: <p>In <a href="https://youtu.be/0bMe_vCZo30?t=779" rel="nofollow noreferrer">a video lecture on the development of neural networks and the history of deep learning</a> (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks stopped until the 80s because people were using the wrong neurons (which were binary so discontinuous) and that is due to the slowness of multiplying floating point numbers which made the use of backpropagation really difficult.</p>
  <p>He said, I quote, &quot;If you have continuous neurons, you need to multiply the activation of a neuron by a weight to get a contribution to the weighted sum.&quot;</p>
  <p>But the statement stays true even with binary (or any discontinuous activation function) neurons. Am I wrong? (at least, as long as you're in the hidden layer, the output of your neuron will be multiplied by a weight I guess). The same professor said that the perceptron, ADALINE relied on weighted sums so they were computing multiplications anyways.</p>
  <p>I don't know what I miss here and I hope someone will enlighten me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T03:03:37.023Z
Comments.Text: Not answering your question directly, but I think a large contribution to the halted development was the lack of computing power. We simply couldn't easily test proposed methods because computers were too slow at the time
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T03:17:29.663Z
Comments.Text: @Recessive thank you for the answer. I'll take it generally that there was a lack of computer power and not exactly the lack of computing power for multiplying floating numbers.
Comments.UserDisplayName: Daviiid
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T03:20:19.653Z
Comments.Text: @DuttaA I agree, but throughout the video it's not said that the weights are 0 or 1. Sure the activation function does outputs zero or one, but the weights can be any real number. I mean the professor said that the perceptron was equipped with potentiometers to vary the weights of the connexions so the weights were real numbers.
Comments.UserDisplayName: Daviiid
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T04:33:02.053Z
Comments.Text: @Daviiid Not necessarily, the GPU was only widely available in early 2000, which is what is primarily used now for fast floating point multiplication, so in the 50's-80's there was an even greater disadvantage to this than other problems.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T17:00:50.267Z
Comments.Text: @DuttaA oh yeah I didn't think about it, thank you !
Comments.UserDisplayName: Daviiid
Tags: <neural-networks><backpropagation><history><weights><perceptron>
Title: Why did the developement of neural networks stop between 50s and 80s?
UserDisplayName: Daviiid
ThreadText: # Why did the developement of neural networks stop between 50s and 80s?
  
  (Asked by: username_0 on 2021-02-24)
  
  In [a video lecture on the development of neural networks and the history of deep learning](https://youtu.be/0bMe_vCZo30?t=779) (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks stopped until the 80s because people were using the wrong neurons (which were binary so discontinuous) and that is due to the slowness of multiplying floating point numbers which made the use of backpropagation really difficult.
  
  He said, I quote, "If you have continuous neurons, you need to multiply the activation of a neuron by a weight to get a contribution to the weighted sum."
  
  But the statement stays true even with binary (or any discontinuous activation function) neurons. Am I wrong? (at least, as long as you're in the hidden layer, the output of your neuron will be multiplied by a weight I guess). The same professor said that the perceptron, ADALINE relied on weighted sums so they were computing multiplications anyways.
  
  I don't know what I miss here and I hope someone will enlighten me.
  
  **Tags:** neural-networks, backpropagation, history, weights, perceptron
  > **Comments:**
  > **username_1**: Not answering your question directly, but I think a large contribution to the halted development was the lack of computing power. We simply couldn't easily test proposed methods because computers were too slow at the time  
  > **username_0**: @Recessive thank you for the answer. I'll take it generally that there was a lack of computer power and not exactly the lack of computing power for multiplying floating numbers.  
  > **username_0**: @DuttaA I agree, but throughout the video it's not said that the weights are 0 or 1. Sure the activation function does outputs zero or one, but the weights can be any real number. I mean the professor said that the perceptron was equipped with potentiometers to vary the weights of the connexions so the weights were real numbers.  
  > **username_1**: @Daviiid Not necessarily, the GPU was only widely available in early 2000, which is what is primarily used now for fast floating point multiplication, so in the 50's-80's there was an even greater disadvantage to this than other problems.  
  > **username_0**: @DuttaA oh yeah I didn't think about it, thank you !  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-11. Score: 1 | Accepted answer
  
  I will first address your main question "Why did the development of neural networks stop between 50s and 80s?"
  In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)!
  
  However, Minsky and Papert have proved in 1969 that a single-layer architecture is not enough to build a universal approximating machine (see e.g. Minsky, M. & Papert, S. Perceptrons: An Introduction to Computational Geometry, vol. 165 (1969)). That led to the first disappointment in "AI". Which lasted until several major breakthroughs in 1980s: the proof of universal approximating capabilities of multi-layer perceptron by Cybenko, the popularisation of backpropagation algorithm (Hinton and colleagues), etc.
  
  I agree with LeCun that using continuous activation functions have enabled the backpropagation algorithm at the time. It is only recently that we have learned to [backpropagate in networks with binary activation functions](https://arxiv.org/pdf/1602.02830.pdf) (2016!).
  
  > **Comments:**
  > **username_0**: Thank you for your answer! LeCunn also says in the same lecture that we can approximate any function with an SVM, is it related to the universal approximating machine you mentioned ? And do you perhaps know in which article it was proved that we can approximate any function with an SVM please ? And that's crazy how it's until recently researchers found a method to train networks with binary activation functions. Like it was theoretically known that backpropagation works with discontinuous functions but we weren't able to do so in practice.  
  > **username_2**: He probably refers to the Universal approximation theorem https://en.wikipedia.org/wiki/Universal_approximation_theorem. You may want to read e.g. this article http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf  
  > **username_0**: Thank you for your clarification and for the article. (I tried to cite you penkovsky with the @ but it's not working I'm sorry)  
  > **username_2**: Happy to help. You may want to accept the answer if you find it satisfying.  
  > **username_0**: Yes that's right, I always forget to do it. Thank you !  
  
  ---
  
  