Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs.  1:4 gives 1:2 on Torch and Caffe, and 1:4 = 1:4 on CNTK.<p>No idea what the catch is :)