1-bit SGD and insanely high minibatch sizes (8192) it would appear: which drastically reduces communication costs, making data-parallel computation scale.<p>If so, while very cool, that&#x27;s not a general solution. Scaling batch sizes of 256 or lower would be the breakthrough.  I suspect they get away with this because speech recognition has very sparse output targets (words&#x2F;phonemes).<p>Too bad the code below isn&#x27;t open-source because they got g2 instances with ~2.5 Gb&#x2F;s interconnect to scale:<p><a href="http:&#x2F;&#x2F;www.nikkostrom.com&#x2F;publications&#x2F;interspeech2015&#x2F;strom_interspeech2015.pdf" rel="nofollow">http:&#x2F;&#x2F;www.nikkostrom.com&#x2F;publications&#x2F;interspeech2015&#x2F;strom...</a>