It makes no sense to make &quot;x10&quot;, or &quot;x40&quot; speedup claims, without mentioning a specific benchmark. 
I&#x27;m guessing the code that scales well with 200 or even 12 GPUs is fairly rare. For example, problems like ImageNet classification with convolutional NNs don&#x27;t scale well beyond 2-4 GPUs. At some point, you will actually get a slowdown, because of the communication overhead.