DeepMind achieves SOTA image recognition with 8.7x faster training
I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google&#x27;s proprietary 100-million image dataset, the model&#x27;s accuracy increases to 87%-whatever.<p>That&#x27;s pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we&#x27;re chasing the imagenet benchmark as if it&#x27;s the holy grail, when in fact it&#x27;s a very narrow slice of what we normally care about as ML researchers. <i>However</i>, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be &quot;just that good now.&quot;<p>Or more succinctly, if the result was bogus, you&#x27;d expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result.<p>The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying.<p>It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it&#x27;s basically a copy-paste replacement.<p>(Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch&#x27;s evonorm-s rather than batchnorm: [1] <a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;sfGVbuq.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;sfGVbuq.png</a> [2] <a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;JMJ1Ll0.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;JMJ1Ll0.png</a> and lol at the fake speedometer.)