Answers.Body: <blockquote>
    <p>If neurons and synapses can be implemented using transistors, </p>
  </blockquote>
  
  <p>I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, Deep Residual Networks, ...). Those were once used as a model for neurons, but they are only <em>very</em> loosely related to what happens in real brain cells.</p>
  
  <p>Spiking networks should be much closer to real neurons. I've heard that the Hodgkin-Huxley model is quite realistic. However - in contrast to the models I named above - there seems not to be an effective training algorithm for spiking networks.</p>
  
  <blockquote>
    <p>what prevents us from creating arbitrarily large neural networks</p>
  </blockquote>
  
  <ul>
  <li><strong>Computational resources</strong>: Training neural networks takes a lot of time. We are talking about ~12 days with a GPU cluster for some CNN models in computer vision.</li>
  <li><strong>Training data</strong>: The more variables you add to the model, the more data you need to estimate those. Neural networks are not magic. They need something they can work with.</li>
  </ul>
  
  <blockquote>
    <p>But the fact is, we today are making silicon chips with billions of transistors on them. SPARK M7 has 10 billion transistors.</p>
    
    <p>If implementing a (non-programmable) neuron and a few hundred synapses for it requires for example 100 000 transistors, then we can make a neural network in hardware that emulates 100 000 neurons.</p>
  </blockquote>
  
  <p>It's not that simple:</p>
  
  <ul>
  <li><strong>Asynchonosity</strong>: Biological neural networks work asynchronously. This means one neuron might be active while all others are not active.</li>
  <li><strong>Emulation</strong>: You assume it would only need one cycle to simulate a biological neuron. However, it needs many thousand cycles. You can't simply use more computational units, because some things are not parallelizable. For example, think of the function <code>f(x) = sin(x*x + 1)</code>. For a human, there are basically three computations: <code>r1 = x*x</code>, <code>r2 = r1 + 1</code>, <code>r3 = sin(r2)</code>. Even if you have 3 people working on calculating the result, you will not be faster than the single fastest person in this group is. Why? Because you need the results of the last computation. </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T16:20:54.740Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105205/discussion-on-answer-by-martin-thoma-are-we-technically-able-to-make-in-hardwar).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-24T05:50:02.653Z
Answers.LastActivityDate: 2016-10-24T05:50:02.653Z
Answers.UserDisplayName: Martin Thoma
Answers.Body: <p>While a single transistor could approximate the basic function of a single neuron, I cannot agree that any electronic element could simulate the synapses/axons. Transistors are etched on a flat surface, and could be interconnected only to adjacent or close by transistors. Axons in the brain span huge distances (compared to the size of the neuron itself), and are not restricted to a two dimensional surface. Even if we were able to approach the number of transistors on a processor to the number of neurons in a brain, we are no where near as number of connections. It could also be argued that the analogue signals in the brain carry more information per unit of time, compared to the binary impulses on a chip. Furthermore, the brain actually have plasticity i.e. connections between neurons can be weakened/discarded or straightened/created, while a CPU cannot do that.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-23T23:55:01.747Z
Answers.Text: Silicon chips are built in many layers, allowing a lot of flexibility on the interconnectivity between neurons. Also, electricity travel quickly - allowing us to share the signal paths by multiplexing. I can imagine several ways to allow high connectivity between both nearby and somewhat distant neurons. Plasticity would be emulated by having a higher degree of connectivity - since a discarded synapse behaves identically to a synapse that multiplies by zero.
Answers.UserDisplayName: frodeborli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-23T21:59:44.617Z
Answers.LastActivityDate: 2016-10-23T21:59:44.617Z
Answers.UserDisplayName: Iliyan Bobev
Answers.Body: <p>You may want to consider this <a href="http://scienceblogs.com/developingintelligence/2007/03/27/why-the-brain-is-not-like-a-co/" rel="nofollow">list</a>:</p>
  
  <blockquote>
    <p>10 important differences between brains and computers:</p>
    
    <ol>
    <li>Brains are analog , computers are digital </li>
    <li>The brain uses content-addressable memory</li>
    <li>The brain is a massively parallel machine; computers are modular and serial </li>
    <li>Processing speed is not fixed in the brain; there is no system clock </li>
    <li>Short-term memory is not like RAM </li>
    <li>No hardware/software distinction can be made with respect to the brain or mind </li>
    <li>Synapses are far more complex than electrical logic gates </li>
    <li>Unlike computers, processing and memory are performed by the same components in the brain </li>
    <li>The brain is a self-organizing system</li>
    <li>Brains have bodies</li>
    </ol>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-23T23:14:13.433Z
Answers.Text: You are making many assumptions and I disagree with many of them. Recurrent neural networks have most of the properties you describe. The body does not have to exist, we merely have to feed the sensory neurons so that this brain belives it has a body - assuming that I want this brain to have a body.
Answers.UserDisplayName: frodeborli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-23T22:04:12.310Z
Answers.LastActivityDate: 2016-10-23T22:04:12.310Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>The approach you describe is called <a href="https://en.wikipedia.org/wiki/Neuromorphic_engineering" rel="noreferrer">neuromorphic computing</a> and it's <a href="https://www.technologyreview.com/s/526506/neuromorphic-chips/" rel="noreferrer">quite</a> a <a href="https://www.uni-heidelberg.de/presse/news2016/pm20160316-neuromorphic-computer-coming-online.html" rel="noreferrer">busy</a> <a href="http://www.nextplatform.com/2016/02/09/the-second-coming-of-neuromorphic-computing/" rel="noreferrer">field</a>. </p>
  
  <p>IBM's <a href="http://www.research.ibm.com/articles/brain-chip.shtml" rel="noreferrer">TrueNorth</a> even has spiking neurons. </p>
  
  <p>The main problem with these projects is that nobody quite knows what to do with them yet. </p>
  
  <p>These projects don't try to create chips that are optimised to <em>run</em> a neural network. That would certainly be possible, but the expensive part is the <em>training</em> not the running of neural networks. And for the training you need huge matrix multiplications, something GPUs are very good at already. (<a href="https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html" rel="noreferrer">Google's TPU</a> would be a chip optimised to run NNs.)</p>
  
  <p>To do research on algorithms that might be implemented in the brain (we hardly know anything about that) you need flexibility, something these chips don't have. Also, the engineering challenge likely lies in providing a lot of synapses, just compare the average number of synapses per neuron of TrueNorth, 256, and the brain, 10,000.</p>
  
  <p>So, you could create a chip designed after some neural architecture and it would be faster, more efficient, etc …, but to do that you'll need to know which architecture works first. We know that deep learning works, so google uses custom made hardware to run their applications and I could certainly imagine custom made deep learning hardware coming to a smartphone near you in the future. To create a neuromorphic chip for strong AI you'd need to develop strong AI first.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-24T11:26:13.650Z
Answers.Text: Thank you. I believe that huge spiking neural networks will behave very intelligently. So if somebody gets an idea (breakthrough) allowing a hardware neuron to connect to 10000 or more neurons efficiently - it would be merely a question having enough silicon to create arbitrarily big neural networks - and perhaps super intelligence. Which implies it could happen today, already have happened in some lab at IBM - or will happen in 20 years. It means we don't have to wait for Moores law.
Answers.UserDisplayName: frodeborli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-24T07:21:32.677Z
Answers.LastActivityDate: 2016-10-24T07:21:32.677Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p><em>If neurons and synapses can be implemented using transistors, what prevents us from creating arbitrarily large neural networks using the same methods with which GPUs are made?</em></p>
  
  <p>In essence, we have seen how extraordinarily well virtual neural networks implemented on sequential processors work (even GPUs are sequential machines, but with huge amounts of cores). </p>
  
  <p>One can imagine that using GPU design principles - which is basically to have thousands of programmable processing units that work in parallel - we could make much simpler "neuron processing units" and put millions or billions of those NPUs in a single big chip. They would have their own memory (for storing weights) and be connected to a few hundred other neurons by sharing a bus. They could have a frequency of for example 20 Hz, which would allow them to share a data bus with many other neurons.</p>
  
  <p>Obviously, there are some electrical engineering challenges here, but it seems to me that all big tech companies should be exploring this route by now.</p>
  
  <p>Many AI researchers say that superintelligence is coming around the year 2045. I believe that their reasoning is based on Moore's law and the number of neurons we are able to implement in software running on the fastest computers we have.</p>
  
  <p>But the fact is, we today are making silicon chips with billions of transistors on them. SPARK M7 has 10 billion transistors.</p>
  
  <p>If implementing a (non-programmable) neuron and a few hundred synapses for it requires for example 100 000 transistors, then we can make a neural network in hardware that emulates 100 000 neurons.</p>
  
  <p>If we design such a chip so that we can simply make it physically bigger if we want more neurons, then it seems to me that arbitrarily large neural networks are simply a budget question.</p>
  
  <p><em>Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?</em></p>
  
  <p>Remember: I am NOT asking if such a network will in fact be very intelligent. I am merely asking if we can factually make arbitrarily large, highly interconnected neural networks, if we decide to pay Intel to do this? </p>
  
  <p>The implication is that on the day some scientist is able to create general intelligence in software, we can use our hardware capabilities to grow this general intelligence to human levels and beyond.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T16:10:18.677Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105204/discussion-on-question-by-frodeborli-are-we-technically-able-to-make-in-hardwar).
Comments.UserDisplayName: nbro
Tags: <neural-networks><recurrent-neural-networks><hardware><implementation>
Title: Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?
UserDisplayName: frodeborli
ThreadText: # Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?
  
  (Asked by: username_0 on 2016-10-23)
  
  *If neurons and synapses can be implemented using transistors, what prevents us from creating arbitrarily large neural networks using the same methods with which GPUs are made?*
  
  In essence, we have seen how extraordinarily well virtual neural networks implemented on sequential processors work (even GPUs are sequential machines, but with huge amounts of cores).
  
  One can imagine that using GPU design principles - which is basically to have thousands of programmable processing units that work in parallel - we could make much simpler "neuron processing units" and put millions or billions of those NPUs in a single big chip. They would have their own memory (for storing weights) and be connected to a few hundred other neurons by sharing a bus. They could have a frequency of for example 20 Hz, which would allow them to share a data bus with many other neurons.
  
  Obviously, there are some electrical engineering challenges here, but it seems to me that all big tech companies should be exploring this route by now.
  
  Many AI researchers say that superintelligence is coming around the year 2045. I believe that their reasoning is based on Moore's law and the number of neurons we are able to implement in software running on the fastest computers we have.
  
  But the fact is, we today are making silicon chips with billions of transistors on them. SPARK M7 has 10 billion transistors.
  
  If implementing a (non-programmable) neuron and a few hundred synapses for it requires for example 100 000 transistors, then we can make a neural network in hardware that emulates 100 000 neurons.
  
  If we design such a chip so that we can simply make it physically bigger if we want more neurons, then it seems to me that arbitrarily large neural networks are simply a budget question.
  
  *Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?*
  
  Remember: I am NOT asking if such a network will in fact be very intelligent. I am merely asking if we can factually make arbitrarily large, highly interconnected neural networks, if we decide to pay Intel to do this?
  
  The implication is that on the day some scientist is able to create general intelligence in software, we can use our hardware capabilities to grow this general intelligence to human levels and beyond.
  
  **Tags:** neural-networks, recurrent-neural-networks, hardware, implementation
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105204/discussion-on-question-by-username_0-are-we-technically-able-to-make-in-hardwar).  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2016-10-24. Score: 7 | Accepted answer
  
  The approach you describe is called [neuromorphic computing](https://en.wikipedia.org/wiki/Neuromorphic_engineering) and it's [quite](https://www.technologyreview.com/s/526506/neuromorphic-chips/) a [busy](https://www.uni-heidelberg.de/presse/news2016/pm20160316-neuromorphic-computer-coming-online.html) [field](http://www.nextplatform.com/2016/02/09/the-second-coming-of-neuromorphic-computing/).
  
  IBM's [TrueNorth](http://www.research.ibm.com/articles/brain-chip.shtml) even has spiking neurons.
  
  The main problem with these projects is that nobody quite knows what to do with them yet.
  
  These projects don't try to create chips that are optimised to *run* a neural network. That would certainly be possible, but the expensive part is the *training* not the running of neural networks. And for the training you need huge matrix multiplications, something GPUs are very good at already. ([Google's TPU](https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html) would be a chip optimised to run NNs.)
  
  To do research on algorithms that might be implemented in the brain (we hardly know anything about that) you need flexibility, something these chips don't have. Also, the engineering challenge likely lies in providing a lot of synapses, just compare the average number of synapses per neuron of TrueNorth, 256, and the brain, 10,000.
  
  So, you could create a chip designed after some neural architecture and it would be faster, more efficient, etc …, but to do that you'll need to know which architecture works first. We know that deep learning works, so google uses custom made hardware to run their applications and I could certainly imagine custom made deep learning hardware coming to a smartphone near you in the future. To create a neuromorphic chip for strong AI you'd need to develop strong AI first.
  
  > **Comments:**
  > **username_0**: Thank you. I believe that huge spiking neural networks will behave very intelligently. So if somebody gets an idea (breakthrough) allowing a hardware neuron to connect to 10000 or more neurons efficiently - it would be merely a question having enough silicon to create arbitrarily big neural networks - and perhaps super intelligence. Which implies it could happen today, already have happened in some lab at IBM - or will happen in 20 years. It means we don't have to wait for Moores law.  
  
  ---
  
  ### Answer by username_2 on 2016-10-24. Score: 4
  
  > If neurons and synapses can be implemented using transistors,
  
  I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, Deep Residual Networks, ...). Those were once used as a model for neurons, but they are only *very* loosely related to what happens in real brain cells.
  
  Spiking networks should be much closer to real neurons. I've heard that the Hodgkin-Huxley model is quite realistic. However - in contrast to the models I named above - there seems not to be an effective training algorithm for spiking networks.
  
  > what prevents us from creating arbitrarily large neural networks
  
  * **Computational resources**: Training neural networks takes a lot of time. We are talking about ~12 days with a GPU cluster for some CNN models in computer vision.
  * **Training data**: The more variables you add to the model, the more data you need to estimate those. Neural networks are not magic. They need something they can work with.
  
  > But the fact is, we today are making silicon chips with billions of transistors on them. SPARK M7 has 10 billion transistors.
  >
  > If implementing a (non-programmable) neuron and a few hundred synapses for it requires for example 100 000 transistors, then we can make a neural network in hardware that emulates 100 000 neurons.
  
  It's not that simple:
  
  * **Asynchonosity**: Biological neural networks work asynchronously. This means one neuron might be active while all others are not active.
  * **Emulation**: You assume it would only need one cycle to simulate a biological neuron. However, it needs many thousand cycles. You can't simply use more computational units, because some things are not parallelizable. For example, think of the function `f(x) = sin(x*x + 1)`. For a human, there are basically three computations: `r1 = x*x`, `r2 = r1 + 1`, `r3 = sin(r2)`. Even if you have 3 people working on calculating the result, you will not be faster than the single fastest person in this group is. Why? Because you need the results of the last computation.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105205/discussion-on-answer-by-username_2-username_2-are-we-technically-able-to-make-in-hardwar).  
  
  ---
  
  ### Answer by username_3 on 2016-10-23. Score: 2
  
  While a single transistor could approximate the basic function of a single neuron, I cannot agree that any electronic element could simulate the synapses/axons. Transistors are etched on a flat surface, and could be interconnected only to adjacent or close by transistors. Axons in the brain span huge distances (compared to the size of the neuron itself), and are not restricted to a two dimensional surface. Even if we were able to approach the number of transistors on a processor to the number of neurons in a brain, we are no where near as number of connections. It could also be argued that the analogue signals in the brain carry more information per unit of time, compared to the binary impulses on a chip. Furthermore, the brain actually have plasticity i.e. connections between neurons can be weakened/discarded or straightened/created, while a CPU cannot do that.
  
  > **Comments:**
  > **username_0**: Silicon chips are built in many layers, allowing a lot of flexibility on the interconnectivity between neurons. Also, electricity travel quickly - allowing us to share the signal paths by multiplexing. I can imagine several ways to allow high connectivity between both nearby and somewhat distant neurons. Plasticity would be emulated by having a higher degree of connectivity - since a discarded synapse behaves identically to a synapse that multiplies by zero.  
  
  ---
  
  ### Answer by username_4 on 2016-10-23. Score: 0
  
  You may want to consider this [list](http://scienceblogs.com/developingintelligence/2007/03/27/why-the-brain-is-not-like-a-co/):
  
  > 10 important differences between brains and computers:
  >
  > 1. Brains are analog , computers are digital
  > 2. The brain uses content-addressable memory
  > 3. The brain is a massively parallel machine; computers are modular and serial
  > 4. Processing speed is not fixed in the brain; there is no system clock
  > 5. Short-term memory is not like RAM
  > 6. No hardware/software distinction can be made with respect to the brain or mind
  > 7. Synapses are far more complex than electrical logic gates
  > 8. Unlike computers, processing and memory are performed by the same components in the brain
  > 9. The brain is a self-organizing system
  > 10. Brains have bodies
  
  > **Comments:**
  > **username_0**: You are making many assumptions and I disagree with many of them. Recurrent neural networks have most of the properties you describe. The body does not have to exist, we merely have to feed the sensory neurons so that this brain belives it has a body - assuming that I want this brain to have a body.  
  
  ---
  
  