<p>If you limited yourself to 12.6 watts, you wouldn't get much done.  Just lookup the power consumption for a modern GPU, look at the size networks people are training on those, and then scale down.  For reference, modern GPU's appear to <a href="http://www.tomshardware.com/reviews/nvidia-geforce-gtx-960,4038-9.html" rel="nofollow">consume between 52-309 watts under heavy use</a>.</p>

<p>Clearly energy efficiency is one area where the human brain is still far head of ANN's.  </p>

CC BY-SA 3.0
2016-09-02T21:03:05.597Z
2016-09-02T21:03:05.597Z
mindcrime
<p><strong>126 million artificial neurons at 12.6 Watts, with IBM's True North</strong></p>

<p>Back in 2014, <a href="http://www.research.ibm.com/articles/brain-chip.shtml">IBM's True North</a> chip was pushing 1 million neurons at less than 100mW.</p>

<p>So that's roughly 126 million artificial neurons at 12.6 Watts.</p>

<p>A <a href="https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons">mouse</a> has 70 million neurons.</p>

<p><a href="http://www.33rdsquare.com/2016/04/ibms-dharmendra-modha-discusses.html">IBM believes</a> they can build a human-brain scale True North mainframe at a "mere" 4kW.</p>

<p>Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.</p>

CC BY-SA 3.0
2016-09-02T22:52:27.460Z
> So that's roughly 126 million artificial neurons at 12.6 Watts.
> A mouse has 70 million neurons.

Problem is, we don't know if a single artificial "neuron" is truly a one to one equivalent for a real biological neuron.
mindcrime
CC BY-SA 3.0
2016-09-02T23:24:13.393Z
In addition to mindcrime, keep in mind that for roughly 60 years, neuroscientists/AI researchers have consistently badly underestimated the processing power of the human brain. I saw early estimates by big names that our minds could only store a few 100 MB! We have yet to factor in things like bit depth, proper architecture, possible quantum states in brain cells, etc. IBM may likely build a power-efficient 70-million-neuron neural net that can't even approximate the behavior of a mouse's parasites. AI researchers have historically *always* badly overestimated their own capabilities.
SQLServerSteve
CC BY-SA 3.0
2016-09-02T23:51:32.650Z
Indeed, forgot about IBM chips. I didn't even take them into consideration when writing the question, as there is known criticism on their efficiency (see e.g. https://www.facebook.com/yann.lecun/posts/10152184295832143). Though technically they still implement neural networks, so +1 from me.
liori
CC BY-SA 3.0
2016-09-03T00:37:22.950Z
@mindcrime & SQLServerSteve Well, the question could have been specifically about vanilla feed forward ANNs running on stock hardware. That's not the impression I got from the question though.
Doxosophoi
CC BY-SA 3.0
2016-09-03T00:48:03.803Z
Nor did it seem to be about artificial neural networks that have yet to be imagined.
Doxosophoi
CC BY-SA 3.0
2016-09-02T21:36:30.487Z
2016-09-02T21:36:30.487Z
Doxosophoi
<p>How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to the human brain energy budget (<a href="http://www.scientificamerican.com/article/thinking-hard-calories/" rel="nofollow noreferrer">12.6 watts</a>)?</p>
<p>Let assume one cycle per second, which seems to roughly match the <a href="http://www.jneurosci.org/content/31/45/16217.full" rel="nofollow noreferrer">firing rate of biological neurons</a>.</p>

<neural-networks>
How big artificial neural networks can we run now if our total energy budget for computation is equivalent to the human brain energy budget?
liori
# How big artificial neural networks can we run now if our total energy budget for computation is equivalent to the human brain energy budget?

(Asked by: username_0 on 2016-09-02)

How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to the human brain energy budget ([12.6 watts](http://www.scientificamerican.com/article/thinking-hard-calories/))?

Let assume one cycle per second, which seems to roughly match the [firing rate of biological neurons](http://www.jneurosci.org/content/31/45/16217.full).

**Tags:** neural-networks


---

## 2 answers

### Answer by username_2 on 2016-09-02. Score: 8 | Accepted answer

**126 million artificial neurons at 12.6 Watts, with IBM's True North**

Back in 2014, [IBM's True North](http://www.research.ibm.com/articles/brain-chip.shtml) chip was pushing 1 million neurons at less than 100mW.

So that's roughly 126 million artificial neurons at 12.6 Watts.

A [mouse](https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons) has 70 million neurons.

[IBM believes](http://www.33rdsquare.com/2016/04/ibms-dharmendra-modha-discusses.html) they can build a human-brain scale True North mainframe at a "mere" 4kW.

Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.

> **Comments:**
> **username_1**: > So that's roughly 126 million artificial neurons at 12.6 Watts. > A mouse has 70 million neurons.  Problem is, we don't know if a single artificial "neuron" is truly a one to one equivalent for a real biological neuron.  
> **username_3**: In addition to username_1, keep in mind that for roughly 60 years, neuroscientists/AI researchers have consistently badly underestimated the processing power of the human brain. I saw early estimates by big names that our minds could only store a few 100 MB! We have yet to factor in things like bit depth, proper architecture, possible quantum states in brain cells, etc. IBM may likely build a power-efficient 70-million-neuron neural net that can't even approximate the behavior of a mouse's parasites. AI researchers have historically *always* badly overestimated their own capabilities.  
> **username_0**: Indeed, forgot about IBM chips. I didn't even take them into consideration when writing the question, as there is known criticism on their efficiency (see e.g. https://www.facebook.com/yann.lecun/posts/10152184295832143). Though technically they still implement neural networks, so +1 from me.  
> **username_2**: @username_1 & username_3 Well, the question could have been specifically about vanilla feed forward ANNs running on stock hardware. That's not the impression I got from the question though.  
> **username_2**: Nor did it seem to be about artificial neural networks that have yet to be imagined.  

---

### Answer by username_1 on 2016-09-02. Score: 3

If you limited yourself to 12.6 watts, you wouldn't get much done. Just lookup the power consumption for a modern GPU, look at the size networks people are training on those, and then scale down. For reference, modern GPU's appear to [consume between 52-309 watts under heavy use](http://www.tomshardware.com/reviews/nvidia-geforce-gtx-960,4038-9.html).

Clearly energy efficiency is one area where the human brain is still far head of ANN's.


---

