Answers.Body: <p>If you limited yourself to 12.6 watts, you wouldn't get much done.  Just lookup the power consumption for a modern GPU, look at the size networks people are training on those, and then scale down.  For reference, modern GPU's appear to <a href="http://www.tomshardware.com/reviews/nvidia-geforce-gtx-960,4038-9.html" rel="nofollow">consume between 52-309 watts under heavy use</a>.</p>
  
  <p>Clearly energy efficiency is one area where the human brain is still far head of ANN's.  </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-02T21:03:05.597Z
Answers.LastActivityDate: 2016-09-02T21:03:05.597Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p><strong>126 million artificial neurons at 12.6 Watts, with IBM's True North</strong></p>
  
  <p>Back in 2014, <a href="http://www.research.ibm.com/articles/brain-chip.shtml">IBM's True North</a> chip was pushing 1 million neurons at less than 100mW.</p>
  
  <p>So that's roughly 126 million artificial neurons at 12.6 Watts.</p>
  
  <p>A <a href="https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons">mouse</a> has 70 million neurons.</p>
  
  <p><a href="http://www.33rdsquare.com/2016/04/ibms-dharmendra-modha-discusses.html">IBM believes</a> they can build a human-brain scale True North mainframe at a "mere" 4kW.</p>
  
  <p>Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-02T22:52:27.460Z
Answers.Text: > So that's roughly 126 million artificial neurons at 12.6 Watts.
  > A mouse has 70 million neurons.
  
  Problem is, we don't know if a single artificial "neuron" is truly a one to one equivalent for a real biological neuron.
Answers.UserDisplayName: mindcrime
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-02T23:24:13.393Z
Answers.Text: In addition to mindcrime, keep in mind that for roughly 60 years, neuroscientists/AI researchers have consistently badly underestimated the processing power of the human brain. I saw early estimates by big names that our minds could only store a few 100 MB! We have yet to factor in things like bit depth, proper architecture, possible quantum states in brain cells, etc. IBM may likely build a power-efficient 70-million-neuron neural net that can't even approximate the behavior of a mouse's parasites. AI researchers have historically *always* badly overestimated their own capabilities.
Answers.UserDisplayName: SQLServerSteve
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-02T23:51:32.650Z
Answers.Text: Indeed, forgot about IBM chips. I didn't even take them into consideration when writing the question, as there is known criticism on their efficiency (see e.g. https://www.facebook.com/yann.lecun/posts/10152184295832143). Though technically they still implement neural networks, so +1 from me.
Answers.UserDisplayName: liori
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-03T00:37:22.950Z
Answers.Text: @mindcrime & SQLServerSteve Well, the question could have been specifically about vanilla feed forward ANNs running on stock hardware. That's not the impression I got from the question though.
Answers.UserDisplayName: Doxosophoi
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-03T00:48:03.803Z
Answers.Text: Nor did it seem to be about artificial neural networks that have yet to be imagined.
Answers.UserDisplayName: Doxosophoi
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-02T21:36:30.487Z
Answers.LastActivityDate: 2016-09-02T21:36:30.487Z
Answers.UserDisplayName: Doxosophoi
Body: <p>How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to the human brain energy budget (<a href="http://www.scientificamerican.com/article/thinking-hard-calories/" rel="nofollow noreferrer">12.6 watts</a>)?</p>
  <p>Let assume one cycle per second, which seems to roughly match the <a href="http://www.jneurosci.org/content/31/45/16217.full" rel="nofollow noreferrer">firing rate of biological neurons</a>.</p>
  
Tags: <neural-networks>
Title: How big artificial neural networks can we run now if our total energy budget for computation is equivalent to the human brain energy budget?
UserDisplayName: liori
ThreadText: # How big artificial neural networks can we run now if our total energy budget for computation is equivalent to the human brain energy budget?
  
  (Asked by: username_0 on 2016-09-02)
  
  How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to the human brain energy budget ([12.6 watts](http://www.scientificamerican.com/article/thinking-hard-calories/))?
  
  Let assume one cycle per second, which seems to roughly match the [firing rate of biological neurons](http://www.jneurosci.org/content/31/45/16217.full).
  
  **Tags:** neural-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2016-09-02. Score: 8 | Accepted answer
  
  **126 million artificial neurons at 12.6 Watts, with IBM's True North**
  
  Back in 2014, [IBM's True North](http://www.research.ibm.com/articles/brain-chip.shtml) chip was pushing 1 million neurons at less than 100mW.
  
  So that's roughly 126 million artificial neurons at 12.6 Watts.
  
  A [mouse](https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons) has 70 million neurons.
  
  [IBM believes](http://www.33rdsquare.com/2016/04/ibms-dharmendra-modha-discusses.html) they can build a human-brain scale True North mainframe at a "mere" 4kW.
  
  Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.
  
  > **Comments:**
  > **username_1**: > So that's roughly 126 million artificial neurons at 12.6 Watts. > A mouse has 70 million neurons.  Problem is, we don't know if a single artificial "neuron" is truly a one to one equivalent for a real biological neuron.  
  > **username_3**: In addition to mindcrime, keep in mind that for roughly 60 years, neuroscientists/AI researchers have consistently badly underestimated the processing power of the human brain. I saw early estimates by big names that our minds could only store a few 100 MB! We have yet to factor in things like bit depth, proper architecture, possible quantum states in brain cells, etc. IBM may likely build a power-efficient 70-million-neuron neural net that can't even approximate the behavior of a mouse's parasites. AI researchers have historically *always* badly overestimated their own capabilities.  
  > **username_0**: Indeed, forgot about IBM chips. I didn't even take them into consideration when writing the question, as there is known criticism on their efficiency (see e.g. https://www.facebook.com/yann.lecun/posts/10152184295832143). Though technically they still implement neural networks, so +1 from me.  
  > **username_2**: @mindcrime & SQLServerSteve Well, the question could have been specifically about vanilla feed forward ANNs running on stock hardware. That's not the impression I got from the question though.  
  > **username_2**: Nor did it seem to be about artificial neural networks that have yet to be imagined.  
  
  ---
  
  ### Answer by username_1 on 2016-09-02. Score: 3
  
  If you limited yourself to 12.6 watts, you wouldn't get much done. Just lookup the power consumption for a modern GPU, look at the size networks people are training on those, and then scale down. For reference, modern GPU's appear to [consume between 52-309 watts under heavy use](http://www.tomshardware.com/reviews/nvidia-geforce-gtx-960,4038-9.html).
  
  Clearly energy efficiency is one area where the human brain is still far head of ANN's.
  
  
  ---
  
  