Answers.Body: <p>The first neural network machine was the <a href="https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinforcement_calculator" rel="nofollow noreferrer"><em>stochastic neural analog reinforcement calculator</em> (SNARC)</a>, built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we did not have back then.</p>
  
  <p>You could look at <a href="https://www.wikiwand.com/en/Timeline_of_machine_learning" rel="nofollow noreferrer">the Wikipedia timeline</a>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-13T10:50:18.570Z
Answers.LastActivityDate: 2019-11-13T12:46:07.087Z
Answers.LastEditDate: 2019-11-13T12:46:07.087Z
Answers.UserDisplayName: Lively
Body: <p>In recent years, we have seen quite a lot of impressive display of Deep Neural Network (DNN), as demonstrated most famously by AlphaGo and its cousin programs.</p>
  <p>But if I understand correctly, <a href="https://ai.stackexchange.com/a/98/16291">deep neural network is just a normal neural network with a lot of layers</a>. We know about the principles of the neural network since the 1970s (?), and a deep neural network is <em>just</em> the generalization of a one-layer neural network to many.</p>
  <p>From here, it doesn't seem like the recent explosion of DNN has anything to do with a <strong>theoretical</strong> breakthrough, such as some new <strong>revolutionary</strong> learning algorithms or particular topologies that have been theoretically proven effective. It seems like DNN successes can be entirely (or mostly) attributed to better hardware and more data, and not to any new theoretical insights or better algorithms.</p>
  <p>I would go even as far as saying that there are no new theoretical insights/algorithms that contribute significantly to the DNN's recent successes; that the most important (if not all) theoretical underpinnings of DNNs were done in the 1970s or prior.</p>
  <p>Am I right on this? How much <strong>weight</strong> (if any) do <strong>theoretical</strong> advancements have in contributing to the recent successes of DNNs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T07:30:06.233Z
Comments.Text: Related: https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye/14387#14387
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T09:50:20.127Z
Comments.Text: @NeilSlater, not quite the same. As I am asking about how much weight  theoretical advancement has ( or is any) in contributing to DNN .
Comments.UserDisplayName: Graviton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T10:27:48.677Z
Comments.Text: What are you willing to count as "theoretical advancement"? It is possible to claim everything is just book-keeping since the invention of calculus and the chain rule (in 17th century). Backpropagation did not invent this theory, just applied it. Optimisation through gradient descent is also a very old theoretically. Neural networks seen in that light are primarily an invention of applied maths/engineering, not theoretical constructs . . . so I would like to understand your threshold for what you consider as a theory?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T10:30:57.200Z
Comments.Text: @Graviton, Several items in the link above can be considered as theoretical advancements. Using ReLU instead of smooth functions, or the theory section above discussing local vs global minima in higher dimensional functions are theoretical; and the introduction of dropout layer is also in the list. But are they revolutionary? That is a different question.
Comments.UserDisplayName: serali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T11:21:18.030Z
Comments.Text: @serali I agree with you that the question whether those advancements are revolutionary is different from the linked SE question, and hence this is *my* question here
Comments.UserDisplayName: Graviton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T11:23:35.957Z
Comments.Text: @NeilSlater my threshold is the same with the scientific community out there . Is there anything revolutionary in terms of theoretical enhancement of DNN as far as the scientific community is concerned?
Comments.UserDisplayName: Graviton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-13T11:32:03.983Z
Comments.Text: That just moves the goalposts. There isn't a "scientific community out there" that has consensus views on how major a theoretical result has to be, before it counts as a significant contribution to the field. It is a matter of personal opinion, unless you set something more objective as a threshold. You could go with papers published with theory in neural networks? In which case there have likely been hundreds since 1970s (including some of the early work defining back-propagation which started in 1970s, but went into the 1980s)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T23:29:46.860Z
Comments.Text: @Graviton To make this question more specific (and less open to subjective interpretation), I suggest that you be more specific. For example, you could say that some (or all) of the info in this answer https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye/14387#14387 does not list "theoretical breakthroughs", for example, you may say that you do not consider the use of ReLU or dropout (or whatever) really a big theoretical breakthrough and that these are just "tricks".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T23:29:51.127Z
Comments.Text: @Graviton Then maybe you can explain what a "breakthrough" is for you with an example. Otherwise, it's not immediately clear what you're really looking for.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><deep-neural-networks><computational-learning-theory>
Title: Is there anything theoretically revolutionary about Deep Neural Networks?
UserDisplayName: Graviton
ThreadText: # Is there anything theoretically revolutionary about Deep Neural Networks?
  
  (Asked by: username_0 on 2019-11-13)
  
  In recent years, we have seen quite a lot of impressive display of Deep Neural Network (DNN), as demonstrated most famously by AlphaGo and its cousin programs.
  
  But if I understand correctly, [deep neural network is just a normal neural network with a lot of layers](https://ai.stackexchange.com/a/98/16291). We know about the principles of the neural network since the 1970s (?), and a deep neural network is *just* the generalization of a one-layer neural network to many.
  
  From here, it doesn't seem like the recent explosion of DNN has anything to do with a **theoretical** breakthrough, such as some new **revolutionary** learning algorithms or particular topologies that have been theoretically proven effective. It seems like DNN successes can be entirely (or mostly) attributed to better hardware and more data, and not to any new theoretical insights or better algorithms.
  
  I would go even as far as saying that there are no new theoretical insights/algorithms that contribute significantly to the DNN's recent successes; that the most important (if not all) theoretical underpinnings of DNNs were done in the 1970s or prior.
  
  Am I right on this? How much **weight** (if any) do **theoretical** advancements have in contributing to the recent successes of DNNs?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, computational-learning-theory
  > **Comments:**
  > **username_1**: Related: https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye/14387#14387  
  > **username_0**: @NeilSlater, not quite the same. As I am asking about how much weight  theoretical advancement has ( or is any) in contributing to DNN .  
  > **username_1**: What are you willing to count as "theoretical advancement"? It is possible to claim everything is just book-keeping since the invention of calculus and the chain rule (in 17th century). Backpropagation did not invent this theory, just applied it. Optimisation through gradient descent is also a very old theoretically. Neural networks seen in that light are primarily an invention of applied maths/engineering, not theoretical constructs . . . so I would like to understand your threshold for what you consider as a theory?  
  > **username_2**: @Graviton, Several items in the link above can be considered as theoretical advancements. Using ReLU instead of smooth functions, or the theory section above discussing local vs global minima in higher dimensional functions are theoretical; and the introduction of dropout layer is also in the list. But are they revolutionary? That is a different question.  
  > **username_0**: @serali I agree with you that the question whether those advancements are revolutionary is different from the linked SE question, and hence this is *my* question here  
  > **username_0**: @NeilSlater my threshold is the same with the scientific community out there . Is there anything revolutionary in terms of theoretical enhancement of DNN as far as the scientific community is concerned?  
  > **username_1**: That just moves the goalposts. There isn't a "scientific community out there" that has consensus views on how major a theoretical result has to be, before it counts as a significant contribution to the field. It is a matter of personal opinion, unless you set something more objective as a threshold. You could go with papers published with theory in neural networks? In which case there have likely been hundreds since 1970s (including some of the early work defining back-propagation which started in 1970s, but went into the 1980s)  
  > **username_3**: @Graviton To make this question more specific (and less open to subjective interpretation), I suggest that you be more specific. For example, you could say that some (or all) of the info in this answer https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye/14387#14387 does not list "theoretical breakthroughs", for example, you may say that you do not consider the use of ReLU or dropout (or whatever) really a big theoretical breakthrough and that these are just "tricks".  
  > **username_3**: @Graviton Then maybe you can explain what a "breakthrough" is for you with an example. Otherwise, it's not immediately clear what you're really looking for.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2019-11-13. Score: 1
  
  The first neural network machine was the [*stochastic neural analog reinforcement calculator* (SNARC)](https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinforcement_calculator), built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we did not have back then.
  
  You could look at [the Wikipedia timeline](https://www.wikiwand.com/en/Timeline_of_machine_learning).
  
  
  ---
  
  