Due to the statements in his thread, is there no speedup expected for classic DNNs @apaszke   ?
I currently run speaker recognition DNN’s with pytorch and increasing the number of GPUs (e.g. 2) used, while at the same time increasing the batchsize ( 256 -> 512 ), does not affect the training time at all. Single GPU training on Switchboard takes me ~300 minutes for a full epoch with a single GPU, as well as with 2 GPUs and doubling the batchsize. The model is a 6 Layer 2048 node DNN. The 2 GPUs are utilized, e.g., are not idle at all.
So for DNN’s there is no speedup expected, if the model parameters are large?